-- Logs begin at Sat 2022-10-01 03:38:53 UTC, end at Sat 2022-10-01 04:37:35 UTC. -- Oct 01 04:03:21.164726 dv-cinder-860044-1-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 01 04:03:21.172263 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sat Oct 1 04:03:21 2022] *** Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: nodename: dv-cinder-860044-1-iscsi Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: machine: x86_64 Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: clock source: unix Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: pcre jit disabled Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: detected number of CPU cores: 4 Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: current working directory: / Oct 01 04:03:21.174928 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: detected binary path: /usr/bin/uwsgi-core Oct 01 04:03:21.175477 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: your processes number limit is 47680 Oct 01 04:03:21.175477 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: your memory page size is 4096 bytes Oct 01 04:03:21.175477 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: detected max file descriptor number: 2048 Oct 01 04:03:21.175477 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: lock engine: pthread robust mutexes Oct 01 04:03:21.175477 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: thunder lock: enabled Oct 01 04:03:21.175477 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 01 04:03:21.175477 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 01 04:03:21.215007 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: Python main interpreter initialized at 0x55d6e727a9c0 Oct 01 04:03:21.215007 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: python threads support enabled Oct 01 04:03:21.215007 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: your server socket listen backlog is limited to 100 connections Oct 01 04:03:21.215007 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: your mercy for graceful operations on workers is 90 seconds Oct 01 04:03:21.215389 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: mapped 403077 bytes (393 KB) for 2 cores Oct 01 04:03:21.215471 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: *** Operational MODE: preforking *** Oct 01 04:03:21.215694 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: *** uWSGI is running in multiple interpreter mode *** Oct 01 04:03:21.215694 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: spawned uWSGI master process (pid: 81575) Oct 01 04:03:21.215910 dv-cinder-860044-1-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 01 04:03:21.216456 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: spawned uWSGI worker 1 (pid: 81586, cores: 1) Oct 01 04:03:21.216791 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: spawned uWSGI worker 2 (pid: 81587, cores: 1) Oct 01 04:03:21.220969 dv-cinder-860044-1-iscsi devstack@keystone.service[81575]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 01 04:03:25.523614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 01 04:03:25.523614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: from cryptography.utils import int_from_bytes Oct 01 04:03:25.524829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 01 04:03:25.524829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: from cryptography.utils import int_from_bytes Oct 01 04:03:25.529260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 01 04:03:25.529260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: from cryptography.utils import int_from_bytes Oct 01 04:03:25.529746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 01 04:03:25.529746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: from cryptography.utils import int_from_bytes Oct 01 04:03:25.575536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 01 04:03:25.575729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 01 04:03:25.575901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] command line args: [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 01 04:03:25.576167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 01 04:03:25.576346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ================================================================================ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 01 04:03:25.576597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] admin_token = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.576842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config_dir = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.577053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.577237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] config_source = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.577416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] debug = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.577615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.578107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.578228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.578396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.578640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.578868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.579046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_config_append = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.579226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.579442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_dir = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.579581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.579688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 01 04:03:25.579813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.579899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 01 04:03:25.580048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.580126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] command line args: [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 01 04:03:25.580292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.580373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 01 04:03:25.580480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ================================================================================ {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 01 04:03:25.580565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.580698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.580779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] admin_token = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.580873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.580970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] config_dir = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] config_source = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] debug = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.581970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] list_limit = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.582995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] log_config_append = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] log_dir = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] log_file = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] publish_errors = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.583871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.584902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] standard_threads = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_journal = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.585890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_json = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.586014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.586089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_stderr = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.586223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] use_syslog = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.586316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.586450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.586526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.586776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.587377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.587522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.587709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.587873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.588122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.588197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.588373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.588519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] publish_errors = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.588597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.588671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.588821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.588896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.589028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.589117 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.589275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.589351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.589511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.589592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.589746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] standard_threads = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.589833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.589981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.590056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.590224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.590303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.590402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.590484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.590579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] use_journal = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.590695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.590775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] use_json = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.590918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] use_stderr = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.590992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.591094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] use_syslog = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.591170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.591324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 01 04:03:25.591400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.591580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.591655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.591822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.591898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.592887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.593986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.594997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.595963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.596870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.597980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598751 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.598976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.599899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600290 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.600837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.601925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.602889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.603905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.604857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.605971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.606975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.607926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.608877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.609981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.610985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.611944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.external = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.612877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.password = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] auth.token = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.613999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.614988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.615929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] auth.external = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.616902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.617878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] auth.password = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.618117 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.618200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] auth.token = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.618523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.618611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.618780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.618931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.619043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.619287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.619371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.619541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.619620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.619775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.619851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] credential.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.620979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.621945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.622982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.623923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.624876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.625984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.626959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.627973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.628913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.629893 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.630997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.631926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.632897 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.633005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.633162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.633240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.633404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.633480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.634226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.634368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.634496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.634623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.634784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.634878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.635971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.636990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.637956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.638892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.639889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.640963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.641965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642655 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.642899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.643991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.644974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.645931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.646952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.647908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.648846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.649039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.649118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.649857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.649950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.650111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.650218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.650359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.650541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.650625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.650799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.650875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.651058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.651219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.651392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.651478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.651646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.651722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.651925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.652000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.652170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.652404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.652482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.652639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.652829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] role.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.652906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.653154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.653291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] role.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.653392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.653622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.653700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.653853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.653929 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.654081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.654330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.654405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.654584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.654821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.654897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.655055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.655287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.655364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.655515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.655726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.655808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.655969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.656191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.656268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.656433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.656579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.656687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.656848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.656944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.657120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.657201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.657405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.657493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.657647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.657893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.657968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.658372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.658601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.658686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.658844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.659992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.660070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.660253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.660330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.660513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.660599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.660764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.660840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.661962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.662979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.663918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.664068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.664145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.664305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.664382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.664538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.664615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.664692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81587) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 01 04:03:25.664809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.665172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.665429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.665673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.665923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.666171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.666420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.666661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.666902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.667139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.667386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.667669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.667904 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.668151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.668419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.668679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.connection = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.668943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.669195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.669438 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.669682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.669921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.670160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.670399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.670638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.670887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.671127 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.671374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.671612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.671861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.672103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.672351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.672583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.672823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.673086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 01 04:03:25.673260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81586) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 01 04:03:25.694333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.694565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.694793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.699119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.699356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.699585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.703555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.707767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.731301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.731530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.735960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.736360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.785699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.785909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.833662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.833893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.835030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.835272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.990594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.990821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.990957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.991859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:25.992076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.014991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.015222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.015450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.015544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.015676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.015796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.015902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.016047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.016134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.016208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.016321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.016402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.016542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.016758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.027460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.027666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.041898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.042035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.042116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.042252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.042331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81587) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.042469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.042686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81586) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 01 04:03:26.059278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.059488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 01 04:03:26.059569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.059712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 01 04:03:26.059825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.060025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 01 04:03:26.060105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.060248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 01 04:03:26.060327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.060525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.060613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.060784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.060862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.061069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.061149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.061334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.070562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.070843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.070923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.071218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.071323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.071542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.071677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.072043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.078531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.079122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.081416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.081703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.081986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.082105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.082272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.082353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.082575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.082656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.082819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.082899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083087 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.083981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.084165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.084244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.084443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.084716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.107199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.108195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.108648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.108945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.109690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.109975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.113579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.113865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.114149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.114652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.114942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.115229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.120583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.120865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.121675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.121963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.126413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.126710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.127000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.127289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.127569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.127660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.127874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.128000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.128133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.128222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.128410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.128496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.128691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.128994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.129269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.129542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.144486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.144765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.145680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.145964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.148603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.149994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.153359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.153662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.153744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.154016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.157883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.158449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.161456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.161669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.164647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.165421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.165588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.165720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.165886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.166099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.166194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.166390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.166474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.166669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.166752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.166949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.182921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.183212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.183468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.183557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.183736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.183835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.184013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.184278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.190698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.191060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.192229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.192586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.193797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.194074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.194156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.194390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.197486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.197705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.197790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.197994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.198077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.198285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.198369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.198568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.198718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.198952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.209815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.209995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.210104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.210282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.210392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.210564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.210679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.210846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.211063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.211224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.211356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.211504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.211668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.212384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.228248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.228380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.231188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.231286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.234061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.234154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.235508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.235696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.236950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.237183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.237274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.237471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.237629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.237875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.244610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.244838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.246095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.246335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.249006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.249251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.249407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.249651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.249734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.249946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.250065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.250233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.257744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.258027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.258132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.258312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.258411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.258632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.265327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81587) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.265591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81586) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 01 04:03:26.265684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.265878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.265971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.266155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.266244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.266432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.266523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.266703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.266794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.266975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.267074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.267249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.267353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.267517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.267617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.267788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.267879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81587) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.268977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.269172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81586) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 01 04:03:26.303617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55d6e727a9c0 pid: 81587 (default app) Oct 01 04:03:26.304848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55d6e727a9c0 pid: 81586 (default app) Oct 01 04:03:26.307427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87845a9e-d16d-418f-ae22-885552e4427b None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:26.307709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87845a9e-d16d-418f-ae22-885552e4427b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:26.307989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87845a9e-d16d-418f-ae22-885552e4427b None None] PATH_INFO: `/v3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:26.312112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1/1] 10.222.0.26 () {50 vars in 791 bytes} [Sat Oct 1 04:03:26 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:33.083542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ccf8be-57c2-4099-bed8-e3a8f8aaf092 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:33.084077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ccf8be-57c2-4099-bed8-e3a8f8aaf092 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:33.084077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ccf8be-57c2-4099-bed8-e3a8f8aaf092 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:33.089015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1/2] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:33 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:03:33.094200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:33.094610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:33.094745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:33.167472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: INFO dbcounter [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] Registered counter for database keystone Oct 01 04:03:33.186449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG oslo_db.sqlalchemy.engines [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81587) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 01 04:03:33.192489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writer thread running {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 01 04:03:33.375269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG passlib.handlers.bcrypt [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81587) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 01 04:03:33.375579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG passlib.handlers.bcrypt [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81587) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 01 04:03:33.435116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:33.436708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:33.567808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 2/3] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:33 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 475 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 01 04:03:33.864441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ccf8be-57c2-4099-bed8-e3a8f8aaf092 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:33.932944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: INFO dbcounter [None req-21ccf8be-57c2-4099-bed8-e3a8f8aaf092 None None] Registered counter for database keystone Oct 01 04:03:33.953932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG oslo_db.sqlalchemy.engines [None req-21ccf8be-57c2-4099-bed8-e3a8f8aaf092 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 01 04:03:33.960104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writer thread running {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 01 04:03:34.081853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ccf8be-57c2-4099-bed8-e3a8f8aaf092 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:34.086034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82663aa4-2986-4b57-95a8-8740080d4b60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:34.087298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82663aa4-2986-4b57-95a8-8740080d4b60 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:34.087694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82663aa4-2986-4b57-95a8-8740080d4b60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:34.087973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82663aa4-2986-4b57-95a8-8740080d4b60 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:34.096234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82663aa4-2986-4b57-95a8-8740080d4b60 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:34.146919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING keystone.common.rbac_enforcer.enforcer [None req-82663aa4-2986-4b57-95a8-8740080d4b60 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 01 04:03:34.147205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82663aa4-2986-4b57-95a8-8740080d4b60 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:34.157144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 2/4] 10.222.0.26 () {58 vars in 1324 bytes} [Sat Oct 1 04:03:33 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:03:34.163243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:34.179874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e02817-830a-4add-ab88-165b3fac113c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:34.185488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8ddad9-59c6-40bb-8339-0da9e6226d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:34.186588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ddad9-59c6-40bb-8339-0da9e6226d60 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:34.186875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ddad9-59c6-40bb-8339-0da9e6226d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:34.187150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8ddad9-59c6-40bb-8339-0da9e6226d60 admin admin] PATH_INFO: `/v3/projects/49c20dd6d2894e7799f891fe5467c7e0` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:34.188734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ddad9-59c6-40bb-8339-0da9e6226d60 admin admin] RBAC: Authorizing `identity:get_project(project_id=49c20dd6d2894e7799f891fe5467c7e0)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:34.241822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING keystone.common.rbac_enforcer.enforcer [None req-5a8ddad9-59c6-40bb-8339-0da9e6226d60 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 01 04:03:34.242176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8ddad9-59c6-40bb-8339-0da9e6226d60 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:34.247634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 3/5] 10.222.0.26 () {56 vars in 1207 bytes} [Sat Oct 1 04:03:34 2022] GET /identity/v3/projects/49c20dd6d2894e7799f891fe5467c7e0 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:36.624437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf404e9-ad58-4ad4-858e-d463af637871 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:36.624958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf404e9-ad58-4ad4-858e-d463af637871 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:36.625140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf404e9-ad58-4ad4-858e-d463af637871 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:36.629054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 3/6] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:36.634891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc1612e-158b-436b-84c0-2558195b98b7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:36.635293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc1612e-158b-436b-84c0-2558195b98b7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:36.635446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc1612e-158b-436b-84c0-2558195b98b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:36.655665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ebc1612e-158b-436b-84c0-2558195b98b7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:36.657206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ebc1612e-158b-436b-84c0-2558195b98b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:36.665515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 4/7] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:36 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 01 04:03:36.929510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf404e9-ad58-4ad4-858e-d463af637871 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:36.943332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf404e9-ad58-4ad4-858e-d463af637871 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:36.948564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cc60af-bffd-4298-b834-c41c479db714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:36.949181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cc60af-bffd-4298-b834-c41c479db714 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:36.949461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cc60af-bffd-4298-b834-c41c479db714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:36.949731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cc60af-bffd-4298-b834-c41c479db714 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:36.957647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cc60af-bffd-4298-b834-c41c479db714 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:36.959434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cc60af-bffd-4298-b834-c41c479db714 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:36.968247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 4/8] 10.222.0.26 () {58 vars in 1324 bytes} [Sat Oct 1 04:03:36 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:03:36.973586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc1612e-158b-436b-84c0-2558195b98b7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:36.982633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc1612e-158b-436b-84c0-2558195b98b7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:36.986920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312fe771-7804-4681-9b1f-21f03e7ea695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:36.987552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312fe771-7804-4681-9b1f-21f03e7ea695 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:36.987842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312fe771-7804-4681-9b1f-21f03e7ea695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:36.988119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312fe771-7804-4681-9b1f-21f03e7ea695 admin admin] PATH_INFO: `/v3/users/280e8a1dffee4eacad394696d3ff53d3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:36.989448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312fe771-7804-4681-9b1f-21f03e7ea695 admin admin] RBAC: Authorizing `identity:get_user(user_id=280e8a1dffee4eacad394696d3ff53d3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:36.991259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312fe771-7804-4681-9b1f-21f03e7ea695 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:36.993266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 5/9] 10.222.0.26 () {56 vars in 1198 bytes} [Sat Oct 1 04:03:36 2022] GET /identity/v3/users/280e8a1dffee4eacad394696d3ff53d3 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:39.194270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40383022-8f2d-4049-b9c4-dc0f4f78cd34 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:39.194826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40383022-8f2d-4049-b9c4-dc0f4f78cd34 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:39.194826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40383022-8f2d-4049-b9c4-dc0f4f78cd34 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:39.196294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 5/10] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:39.201549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2282b38-3b6d-4319-a5cc-0e22d544c176 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:39.201968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2282b38-3b6d-4319-a5cc-0e22d544c176 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:39.202101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2282b38-3b6d-4319-a5cc-0e22d544c176 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:39.220283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b2282b38-3b6d-4319-a5cc-0e22d544c176 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:39.221505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b2282b38-3b6d-4319-a5cc-0e22d544c176 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:39.250208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25279cf2-dd37-4c61-b0c9-ec1891ccba3f None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:39.250501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25279cf2-dd37-4c61-b0c9-ec1891ccba3f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:39.250781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25279cf2-dd37-4c61-b0c9-ec1891ccba3f None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:39.252097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 6/11] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:03:39.257264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:39.257549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:39.257830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:39.299386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG passlib.handlers.bcrypt [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81586) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 01 04:03:39.299773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG passlib.handlers.bcrypt [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81586) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 01 04:03:39.303139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 6/12] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:39.349661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:39.351231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:39.451398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 7/13] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:03:39.575213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2282b38-3b6d-4319-a5cc-0e22d544c176 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:39.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:39.679899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2282b38-3b6d-4319-a5cc-0e22d544c176 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:39.713018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:39.713611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:39.713897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:39.714248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:39.715659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:39.725419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:03:39.734768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:39.735869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:03:39.736694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 7/14] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:03:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:03:39.741515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:39.819521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d22c2-bccc-4174-b435-0a937e9a7273 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:39.822458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:39.825243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:39.845129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596dd9d1-388b-4ca0-934a-3976a6434e01 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:39.859492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:39.860790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:39.861319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:39.861657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:39.880645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:39.881282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:39.881567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:39.881842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:39.882523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:39.882834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:39.884744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:39.896505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 8/15] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:03:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:39.902603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:39.928484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:39.930035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:40.004797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5817397-4d54-4277-b1a2-d72ebde27471 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:40.030585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 8/16] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:03:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 01 04:03:40.036373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:40.038662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:40.039680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:40.039792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:40.039865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:40.040554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] PATH_INFO: `/v3/users/280e8a1dffee4eacad394696d3ff53d3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:40.041743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] RBAC: Authorizing `identity:get_user(user_id=280e8a1dffee4eacad394696d3ff53d3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:40.043195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:40.044745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 9/17] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:03:39 2022] GET /identity/v3/users/280e8a1dffee4eacad394696d3ff53d3 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:40.050165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:40.065452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:40.135839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10899fa-6b3c-4b8b-b721-5674cfd44fc9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:40.150392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c05583-25c3-4dd1-b1df-06e58cee6ac2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:40.170609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:40.171458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:40.171739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:40.172011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:40.183466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:40.184108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:40.184391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:40.184667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:40.185813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:40.187262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:40.189089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 10/18] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:40.189926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:40.191371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:40.194505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:03:40.209402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:40.210620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:03:40.211943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:03:40.211943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:03:40.211943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:03:40.211943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 9/19] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:03:40.217030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:40.273470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:40.277080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:40.295994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07d9e18-6503-45e7-9ad4-2299e9ed3617 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:40.324780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781a82b5-b821-467d-991c-4af0d38c568f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:40.331900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bb0265-cff6-4b54-93a2-6bd9bfb02384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:40.332516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bb0265-cff6-4b54-93a2-6bd9bfb02384 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:40.332796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bb0265-cff6-4b54-93a2-6bd9bfb02384 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:40.333109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bb0265-cff6-4b54-93a2-6bd9bfb02384 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:40.333829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bb0265-cff6-4b54-93a2-6bd9bfb02384 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.domain.id=default, user.id=280e8a1dffee4eacad394696d3ff53d3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:40.334137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bb0265-cff6-4b54-93a2-6bd9bfb02384 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:40.335609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bb0265-cff6-4b54-93a2-6bd9bfb02384 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:40.343735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 11/20] 10.222.0.26 () {56 vars in 1322 bytes} [Sat Oct 1 04:03:40 2022] GET /identity/v3/role_assignments?user.id=280e8a1dffee4eacad394696d3ff53d3&scope.domain.id=default&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 233 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:40.358750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07840d6a-931f-43bb-9361-bed65b9de803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:40.359419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07840d6a-931f-43bb-9361-bed65b9de803 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:40.359701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07840d6a-931f-43bb-9361-bed65b9de803 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:40.359974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07840d6a-931f-43bb-9361-bed65b9de803 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:40.360641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07840d6a-931f-43bb-9361-bed65b9de803 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:40.361065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07840d6a-931f-43bb-9361-bed65b9de803 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:40.362508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07840d6a-931f-43bb-9361-bed65b9de803 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:40.382146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 10/21] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:03:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:42.793209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7856df-7a60-41aa-829c-094804369aee None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:42.793681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7856df-7a60-41aa-829c-094804369aee None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:42.793681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7856df-7a60-41aa-829c-094804369aee None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:42.795217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 12/22] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:42.800492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01eebe9-ea5f-4e19-9d51-b536ebc734d2 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:42.801340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01eebe9-ea5f-4e19-9d51-b536ebc734d2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:42.801819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01eebe9-ea5f-4e19-9d51-b536ebc734d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:42.829422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d01eebe9-ea5f-4e19-9d51-b536ebc734d2 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:42.831007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d01eebe9-ea5f-4e19-9d51-b536ebc734d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:42.921318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 11/23] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:43.048977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51580f26-0f45-4037-900f-2ffe57637fc6 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.049607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51580f26-0f45-4037-900f-2ffe57637fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.050079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51580f26-0f45-4037-900f-2ffe57637fc6 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.052057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 13/24] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:43.071758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099c80a-bf40-4407-8c01-042a4c36bb42 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.072066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099c80a-bf40-4407-8c01-042a4c36bb42 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.072326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099c80a-bf40-4407-8c01-042a4c36bb42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.101896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0099c80a-bf40-4407-8c01-042a4c36bb42 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:43.103362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0099c80a-bf40-4407-8c01-042a4c36bb42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:43.152013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:43.187189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 12/25] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:43.418537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9526a3-abbe-4a7b-b353-cf7cebae6ae2 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.418869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9526a3-abbe-4a7b-b353-cf7cebae6ae2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.419050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9526a3-abbe-4a7b-b353-cf7cebae6ae2 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.420495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 14/26] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:43.438455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9967a62-728d-4a3d-8ef0-25ddc89029c2 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.438736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9967a62-728d-4a3d-8ef0-25ddc89029c2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.439014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9967a62-728d-4a3d-8ef0-25ddc89029c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.439488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680caf8-b23a-44d2-a190-ae23a4bd85e1 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.439973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680caf8-b23a-44d2-a190-ae23a4bd85e1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.440474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680caf8-b23a-44d2-a190-ae23a4bd85e1 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.442388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 15/27] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:43.453992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259ce2e1-619c-4462-b390-44480caf5ce5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.454522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259ce2e1-619c-4462-b390-44480caf5ce5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.455035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259ce2e1-619c-4462-b390-44480caf5ce5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.462726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c9967a62-728d-4a3d-8ef0-25ddc89029c2 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:43.463909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c9967a62-728d-4a3d-8ef0-25ddc89029c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:43.480151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-259ce2e1-619c-4462-b390-44480caf5ce5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:43.481652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-259ce2e1-619c-4462-b390-44480caf5ce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:43.600874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 13/28] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:43.601956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 16/29] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:43.602816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259ce2e1-619c-4462-b390-44480caf5ce5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:43.603226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9967a62-728d-4a3d-8ef0-25ddc89029c2 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:43.718010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9967a62-728d-4a3d-8ef0-25ddc89029c2 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:43.725388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:43.756271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:43.757042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.757390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.757725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.773396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:03:43.781884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:43.782793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:03:43.783827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 14/30] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:03:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:03:43.790772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:43.820708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:43.860722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259ce2e1-619c-4462-b390-44480caf5ce5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:43.900587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:43.901260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:43.901639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:43.901939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:43.921608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:43.923561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:03:43.940339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:43.941233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:03:43.942024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:03:43.942024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:03:43.942024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:03:43.942024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 17/31] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:03:43.943186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:43.994398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d766ee96-e206-4a8f-a282-52e517afb7d6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:44.039059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:44.040076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.040364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.040639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.041367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:44.041677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:44.044081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:44.073419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 15/32] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:03:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:44.076511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:44.112820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:44.118717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576764f9-bb25-456d-a491-2c4f8b7acf22 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:44.152605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:44.166193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:44.167047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.167472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.168294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.169155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:44.171064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:44.197568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fcd81a-d956-468d-ab58-cb4f34fb370f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:44.234183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:44.234900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.235288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.235620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.236366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:44.236731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:44.238335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:44.256759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 16/33] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:03:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:44.256759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:44.336956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 18/34] 10.222.0.26 () {60 vars in 1131 bytes} [Sat Oct 1 04:03:43 2022] POST /identity/v3/roles => generated 219 bytes in 388 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:03:44.336956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:44.380260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:44.398429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a5f44-2807-4301-b38f-7e27e585f451 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:44.469632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:44.470732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.472025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.473248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] PATH_INFO: `/v3/users/280e8a1dffee4eacad394696d3ff53d3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.475253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] RBAC: Authorizing `identity:get_user(user_id=280e8a1dffee4eacad394696d3ff53d3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:44.478372 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:44.481148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 17/35] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:03:44 2022] GET /identity/v3/users/280e8a1dffee4eacad394696d3ff53d3 => generated 247 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:44.482801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:44.577086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602eaeb2-3986-4aff-b622-23500f815271 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:44.617056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6799b2b-cdf7-403d-89ba-ee91cf3883a1 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:44.620207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:44.620871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.621186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.621465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.622967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:44.623843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:44.646845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 19/36] 10.222.0.26 () {60 vars in 1131 bytes} [Sat Oct 1 04:03:44 2022] POST /identity/v3/roles => generated 225 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:03:44.647899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:44.669742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f138ea04-40e5-4f21-b945-c2c9657cc05a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:44.670388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f138ea04-40e5-4f21-b945-c2c9657cc05a None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.670673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f138ea04-40e5-4f21-b945-c2c9657cc05a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.670949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f138ea04-40e5-4f21-b945-c2c9657cc05a None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.671628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f138ea04-40e5-4f21-b945-c2c9657cc05a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:44.677453 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f138ea04-40e5-4f21-b945-c2c9657cc05a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:44.731540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:44.810627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b12903c-eb90-4b25-8d9b-ff21e9e112e8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:44.833678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 18/37] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:03:44 2022] POST /identity/v3/projects => generated 301 bytes in 351 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:03:44.836446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f11dc12-4b34-451b-9d3e-c8f289232e98 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.837510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f11dc12-4b34-451b-9d3e-c8f289232e98 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.838004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f11dc12-4b34-451b-9d3e-c8f289232e98 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.842667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 19/38] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:44.842667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4a0849-46cf-4d83-aad4-b6489d82ce21 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.842667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4a0849-46cf-4d83-aad4-b6489d82ce21 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.842667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4a0849-46cf-4d83-aad4-b6489d82ce21 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.842667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 20/39] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:44.849248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd269bc7-de58-4207-8eb2-ec5649c3ba2e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd269bc7-de58-4207-8eb2-ec5649c3ba2e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.850772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd269bc7-de58-4207-8eb2-ec5649c3ba2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.860693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaab858-a7f2-4ef0-baec-f275ce26a406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:44.861571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaab858-a7f2-4ef0-baec-f275ce26a406 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.861955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaab858-a7f2-4ef0-baec-f275ce26a406 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.862363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaab858-a7f2-4ef0-baec-f275ce26a406 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.863877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaab858-a7f2-4ef0-baec-f275ce26a406 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:44.865552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaab858-a7f2-4ef0-baec-f275ce26a406 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:44.868393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 20/40] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:44.870049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050cad7f-1889-4d1b-bc53-4a9820e59ea0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:44.870342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050cad7f-1889-4d1b-bc53-4a9820e59ea0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:44.870623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050cad7f-1889-4d1b-bc53-4a9820e59ea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:44.876058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-bd269bc7-de58-4207-8eb2-ec5649c3ba2e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:44.877852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-bd269bc7-de58-4207-8eb2-ec5649c3ba2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:44.880271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:44.901023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-050cad7f-1889-4d1b-bc53-4a9820e59ea0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:44.902652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-050cad7f-1889-4d1b-bc53-4a9820e59ea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:44.982919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 21/41] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:44.984200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd269bc7-de58-4207-8eb2-ec5649c3ba2e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:45.100762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 21/42] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:45.104263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd269bc7-de58-4207-8eb2-ec5649c3ba2e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:45.141652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:45.142320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:45.142602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:45.142881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] PATH_INFO: `/v3/domains/default/users/280e8a1dffee4eacad394696d3ff53d3/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:45.144791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=280e8a1dffee4eacad394696d3ff53d3, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:45.146379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:45.157895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:45.163919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 22/43] 10.222.0.26 () {58 vars in 1361 bytes} [Sat Oct 1 04:03:44 2022] PUT /identity/v3/domains/default/users/280e8a1dffee4eacad394696d3ff53d3/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:03:45.280966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050cad7f-1889-4d1b-bc53-4a9820e59ea0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:45.331724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:45.371215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:45.393328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050cad7f-1889-4d1b-bc53-4a9820e59ea0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:45.431416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:45.432046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:45.432330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:45.432603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:45.433310 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:45.435241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:45.451833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 22/44] 10.222.0.26 () {60 vars in 1131 bytes} [Sat Oct 1 04:03:45 2022] POST /identity/v3/roles => generated 223 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:03:45.492856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9410a4bc-cde7-4c7b-89df-285a9b9199f5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:45.534773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1f166d-29cc-4992-bfea-47da65fd563c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:45.535443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1f166d-29cc-4992-bfea-47da65fd563c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:45.535726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1f166d-29cc-4992-bfea-47da65fd563c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:45.536000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1f166d-29cc-4992-bfea-47da65fd563c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:45.537211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1f166d-29cc-4992-bfea-47da65fd563c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:45.538688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1f166d-29cc-4992-bfea-47da65fd563c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:45.540726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 23/45] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:45.546391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:45.655187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ce58f0-fe0c-4484-bb5f-aab5a337138a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:45.657988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:45.691879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ca7dd1-f6da-46b1-b931-66624fb79965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:45.692500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ca7dd1-f6da-46b1-b931-66624fb79965 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:45.692786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ca7dd1-f6da-46b1-b931-66624fb79965 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:45.693119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ca7dd1-f6da-46b1-b931-66624fb79965 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:45.693778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ca7dd1-f6da-46b1-b931-66624fb79965 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:45.695273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ca7dd1-f6da-46b1-b931-66624fb79965 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:45.727283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 23/46] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:03:45 2022] POST /identity/v3/projects => generated 312 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:03:47.370337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cfd26f-8c78-42e7-bd51-98d8850149d6 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:47.370886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cfd26f-8c78-42e7-bd51-98d8850149d6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:47.370886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cfd26f-8c78-42e7-bd51-98d8850149d6 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:47.372390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 24/47] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:47.377437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa19722-cd4b-49fa-9171-c00ac9b57e31 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:47.377815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa19722-cd4b-49fa-9171-c00ac9b57e31 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:47.378108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa19722-cd4b-49fa-9171-c00ac9b57e31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:47.396585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4fa19722-cd4b-49fa-9171-c00ac9b57e31 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:47.397788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4fa19722-cd4b-49fa-9171-c00ac9b57e31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:47.463312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 24/48] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:03:47.729696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cfd26f-8c78-42e7-bd51-98d8850149d6 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:47.796624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:47.833433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cfd26f-8c78-42e7-bd51-98d8850149d6 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:47.866307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d8ceaf-f9c8-42a7-96a8-bfddc908cdf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:47.866934 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d8ceaf-f9c8-42a7-96a8-bfddc908cdf6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:47.867215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d8ceaf-f9c8-42a7-96a8-bfddc908cdf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:47.868180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d8ceaf-f9c8-42a7-96a8-bfddc908cdf6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:47.879123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d8ceaf-f9c8-42a7-96a8-bfddc908cdf6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-38d8ceaf-f9c8-42a7-96a8-bfddc908cdf6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:03:47.883969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:47.884788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:03:47.886922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 25/49] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:03:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:03:47.890908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa19722-cd4b-49fa-9171-c00ac9b57e31 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:47.911389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16eb42-50c5-410c-8d45-7f7f5ac9e150 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:47.911675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16eb42-50c5-410c-8d45-7f7f5ac9e150 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:47.911948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d16eb42-50c5-410c-8d45-7f7f5ac9e150 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:47.913289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 26/50] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:03:47.919277 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad2d942-5feb-4e94-9e89-a523a9b2ce90 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:47.919556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad2d942-5feb-4e94-9e89-a523a9b2ce90 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:47.919834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad2d942-5feb-4e94-9e89-a523a9b2ce90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:47.938665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-bad2d942-5feb-4e94-9e89-a523a9b2ce90 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:47.939873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-bad2d942-5feb-4e94-9e89-a523a9b2ce90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:47.975488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:47.992903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa19722-cd4b-49fa-9171-c00ac9b57e31 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:48.007926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 27/51] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:03:48.027612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:48.028233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:48.028521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:48.028798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:48.029515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:48.029828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:48.031248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:48.039532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 25/52] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:03:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:48.045342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad2d942-5feb-4e94-9e89-a523a9b2ce90 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:48.211808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:48.254659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad2d942-5feb-4e94-9e89-a523a9b2ce90 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:48.287435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:48.288069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:48.288347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:48.288618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] PATH_INFO: `/v3/users/280e8a1dffee4eacad394696d3ff53d3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:48.289573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:48.289912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] RBAC: Authorizing `identity:get_user(user_id=280e8a1dffee4eacad394696d3ff53d3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:48.291314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:48.292871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 28/53] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:03:48 2022] GET /identity/v3/users/280e8a1dffee4eacad394696d3ff53d3 => generated 247 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:48.298199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:48.390381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5c372f-90bb-4dc7-81ac-135350ebd401 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:48.395434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a5c4c6-b910-4c46-8195-8938baa44160 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:48.418652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:48.427141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:48.427784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:48.428068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:48.428345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:48.429480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:48.430914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a489b719-978f-4751-a91b-e90656800cba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:48.431464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:48.431633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a489b719-978f-4751-a91b-e90656800cba None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:48.431921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a489b719-978f-4751-a91b-e90656800cba None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:48.432198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a489b719-978f-4751-a91b-e90656800cba None admin] PATH_INFO: `/v3/domains/default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:48.433041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 26/54] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:48.433359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a489b719-978f-4751-a91b-e90656800cba None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:48.434790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a489b719-978f-4751-a91b-e90656800cba None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:48.436344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 29/55] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:03:48.438543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:48.441823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a489b719-978f-4751-a91b-e90656800cba None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:48.457573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:48.541045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a489b719-978f-4751-a91b-e90656800cba None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:48.542248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971b7f0-f97a-437e-8925-9f4641be7d00 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:48.574060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2056f01-6ca4-48df-9ff4-75a3c41691c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:48.574670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2056f01-6ca4-48df-9ff4-75a3c41691c3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:48.574963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2056f01-6ca4-48df-9ff4-75a3c41691c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:48.575239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2056f01-6ca4-48df-9ff4-75a3c41691c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:48.575548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363a9b2c-7d90-49fa-9999-b8f771de8236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:48.575963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2056f01-6ca4-48df-9ff4-75a3c41691c3 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.domain.id=default, user.id=280e8a1dffee4eacad394696d3ff53d3) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:48.576190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363a9b2c-7d90-49fa-9999-b8f771de8236 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:48.576282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2056f01-6ca4-48df-9ff4-75a3c41691c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:48.576481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363a9b2c-7d90-49fa-9999-b8f771de8236 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:48.576764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363a9b2c-7d90-49fa-9999-b8f771de8236 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:48.577463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363a9b2c-7d90-49fa-9999-b8f771de8236 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:48.577734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2056f01-6ca4-48df-9ff4-75a3c41691c3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:48.578899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363a9b2c-7d90-49fa-9999-b8f771de8236 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:48.585984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 30/56] 10.222.0.26 () {56 vars in 1322 bytes} [Sat Oct 1 04:03:48 2022] GET /identity/v3/role_assignments?user.id=280e8a1dffee4eacad394696d3ff53d3&scope.domain.id=default&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 530 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:48.608054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 27/57] 10.222.0.26 () {60 vars in 1140 bytes} [Sat Oct 1 04:03:48 2022] POST /identity/v3/projects => generated 298 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:03:50.778493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba8791-acc3-4ba3-8877-8e22b9585237 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:50.778989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba8791-acc3-4ba3-8877-8e22b9585237 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:50.778989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba8791-acc3-4ba3-8877-8e22b9585237 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:50.780310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 31/58] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:50.785799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5536bc8-2a6e-441c-be5e-b1f5af64a59a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:50.786194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5536bc8-2a6e-441c-be5e-b1f5af64a59a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:50.786374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5536bc8-2a6e-441c-be5e-b1f5af64a59a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:50.804778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b5536bc8-2a6e-441c-be5e-b1f5af64a59a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:50.806667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b5536bc8-2a6e-441c-be5e-b1f5af64a59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:50.865131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:50.871698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 28/59] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:51.141423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ba8791-acc3-4ba3-8877-8e22b9585237 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:51.199862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:51.243980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ba8791-acc3-4ba3-8877-8e22b9585237 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:51.276659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566a1c02-9b57-4888-815b-ac06ac43b8b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:51.277315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566a1c02-9b57-4888-815b-ac06ac43b8b1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:51.277605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566a1c02-9b57-4888-815b-ac06ac43b8b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:51.277881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566a1c02-9b57-4888-815b-ac06ac43b8b1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:51.278981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566a1c02-9b57-4888-815b-ac06ac43b8b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:51.280509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566a1c02-9b57-4888-815b-ac06ac43b8b1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:51.282084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 32/60] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:51.287730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5536bc8-2a6e-441c-be5e-b1f5af64a59a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:51.382346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5536bc8-2a6e-441c-be5e-b1f5af64a59a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:51.421702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54beb85-83ec-4156-b21e-850f81e51bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:51.422322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54beb85-83ec-4156-b21e-850f81e51bca None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:51.422608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54beb85-83ec-4156-b21e-850f81e51bca None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:51.422882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54beb85-83ec-4156-b21e-850f81e51bca None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:51.423542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54beb85-83ec-4156-b21e-850f81e51bca None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:51.425016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54beb85-83ec-4156-b21e-850f81e51bca None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:51.452192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 29/61] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:03:51 2022] POST /identity/v3/users => generated 275 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:03:54.727405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf5731-10fc-477d-bf21-137c99b589c9 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:54.728404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf5731-10fc-477d-bf21-137c99b589c9 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:54.728920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf5731-10fc-477d-bf21-137c99b589c9 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:54.730782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 33/62] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:54.735829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbe6a50-76ec-4392-ad60-b897fdcd80c7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:54.736669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbe6a50-76ec-4392-ad60-b897fdcd80c7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:54.737177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbe6a50-76ec-4392-ad60-b897fdcd80c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:54.757057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-dfbe6a50-76ec-4392-ad60-b897fdcd80c7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:54.759239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dfbe6a50-76ec-4392-ad60-b897fdcd80c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:54.833942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:54.862445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 30/63] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:55.015355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da47f6b-ae7e-4c31-a694-421a4ffbcc44 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.016026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da47f6b-ae7e-4c31-a694-421a4ffbcc44 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.016509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da47f6b-ae7e-4c31-a694-421a4ffbcc44 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.018546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 34/64] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:55.035054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2f5ce3-27c8-4049-9136-9ad738354ce5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.035337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2f5ce3-27c8-4049-9136-9ad738354ce5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.035789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2f5ce3-27c8-4049-9136-9ad738354ce5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.066158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cc2f5ce3-27c8-4049-9136-9ad738354ce5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:55.067604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cc2f5ce3-27c8-4049-9136-9ad738354ce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:55.159802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da47f6b-ae7e-4c31-a694-421a4ffbcc44 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:55.180482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 31/65] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:55.212887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e96aa2-7fb9-4de1-acd4-cc80fd869dfb None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.213224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e96aa2-7fb9-4de1-acd4-cc80fd869dfb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.213487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e96aa2-7fb9-4de1-acd4-cc80fd869dfb None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.215086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 32/66] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:55.230904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5592930a-90cb-4da9-89ff-12fbfd051863 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.231196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5592930a-90cb-4da9-89ff-12fbfd051863 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.231481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5592930a-90cb-4da9-89ff-12fbfd051863 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.258336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5592930a-90cb-4da9-89ff-12fbfd051863 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:55.259560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5592930a-90cb-4da9-89ff-12fbfd051863 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:55.321967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da47f6b-ae7e-4c31-a694-421a4ffbcc44 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:55.351403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 33/67] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:03:55.359064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49cb56-d01b-4b5a-a006-26af490f4c79 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.359635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49cb56-d01b-4b5a-a006-26af490f4c79 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.360056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:55.360269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49cb56-d01b-4b5a-a006-26af490f4c79 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.362226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 34/68] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:55.369677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffffbcc-6fbb-4e87-90fe-d8d1e7f82b00 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.370306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffffbcc-6fbb-4e87-90fe-d8d1e7f82b00 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.370565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:55.371020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffffbcc-6fbb-4e87-90fe-d8d1e7f82b00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.371219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.371507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.371786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.383614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:03:55.389363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:55.390382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:03:55.391327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:03:55.394125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 35/69] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:03:55.406044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:55.406236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4ffffbcc-6fbb-4e87-90fe-d8d1e7f82b00 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:55.407532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4ffffbcc-6fbb-4e87-90fe-d8d1e7f82b00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:55.482647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:55.516416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 35/70] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:55.517684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffffbcc-6fbb-4e87-90fe-d8d1e7f82b00 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:55.533954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72308fb7-fd3b-4d1e-a1f0-4a6f95a468de None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:55.581354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab855647-5a30-493c-9657-ecc1e6c39e6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:55.581974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab855647-5a30-493c-9657-ecc1e6c39e6a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.582260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab855647-5a30-493c-9657-ecc1e6c39e6a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.582539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab855647-5a30-493c-9657-ecc1e6c39e6a None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.583214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab855647-5a30-493c-9657-ecc1e6c39e6a None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:55.583524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab855647-5a30-493c-9657-ecc1e6c39e6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:55.584911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab855647-5a30-493c-9657-ecc1e6c39e6a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:55.598539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 36/71] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:55.600019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1d99b2-7c54-4a69-a2b9-f338d0b070f3 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.600550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1d99b2-7c54-4a69-a2b9-f338d0b070f3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.601044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1d99b2-7c54-4a69-a2b9-f338d0b070f3 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.603212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 37/72] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:55.607622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1d99b2-7c54-4a69-a2b9-f338d0b070f3 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:55.657608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffffbcc-6fbb-4e87-90fe-d8d1e7f82b00 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:55.667365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:55.692641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd813d8f-9f9f-4888-acec-58d671d2f7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:55.693399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd813d8f-9f9f-4888-acec-58d671d2f7b9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.693688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd813d8f-9f9f-4888-acec-58d671d2f7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.693959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd813d8f-9f9f-4888-acec-58d671d2f7b9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.705491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd813d8f-9f9f-4888-acec-58d671d2f7b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-fd813d8f-9f9f-4888-acec-58d671d2f7b9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:03:55.710347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:55.711269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 36/73] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:03:55.713805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c84490b-0272-423b-a393-b4793384b722 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.714561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c84490b-0272-423b-a393-b4793384b722 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.714561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c84490b-0272-423b-a393-b4793384b722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.730001 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1d99b2-7c54-4a69-a2b9-f338d0b070f3 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:55.737226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0c84490b-0272-423b-a393-b4793384b722 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:55.738356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0c84490b-0272-423b-a393-b4793384b722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:55.770768 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:55.771013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:55.771408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.771720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.772038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.790847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:55.792309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:55.794100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 38/74] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:55.794864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:55.816155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 37/75] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:03:55.817274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c84490b-0272-423b-a393-b4793384b722 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:55.901689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27afaced-80b4-40e5-9668-72219c191f40 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:55.904631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:55.927205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c84490b-0272-423b-a393-b4793384b722 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:55.940102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:55.940735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.941046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.941325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.951792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:03:55.957189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:55.958036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:03:55.958790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 39/76] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:03:55.959428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:55.966471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:55.967364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:55.967863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:55.968332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:55.969241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:55.969744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:55.971453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:55.980827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 38/77] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:55.981986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.010452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.077869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c6542-744b-4c22-8b9f-89a5bc99fc02 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.091896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580d89aa-7a64-43b9-ab26-6eda5f2c908e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.112668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.113320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.113605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.113879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.124503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:03:56.126186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.126800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.127087 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.127367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:03:56.129268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:03:56.130125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:03:56.130938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 40/78] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:03:56.131569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.157048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.200125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.201623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.226210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.231068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb90bff-caf2-4bc9-a6fb-184df57b2450 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.264412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.264967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.265278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.265556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.266218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:56.266525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.267902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.275741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 41/79] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:56.276533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.309655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 39/80] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:03:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:03:56.309655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.360503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.376249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a08ce6-1f78-4f5d-a443-b4296a33907d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.408998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6018d0b7-b787-4bd5-ba95-c64aea0b4757 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.411667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.412303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.412590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.412869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.414039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.416578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.418192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 42/81] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:56.418955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.439265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.448334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.448998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.449289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.449591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.450703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.452153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.453740 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 40/82] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:56.454497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.522535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b65ded-d8be-4034-b11f-497b8897ee46 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.556364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370c2b57-5a8f-49c0-b332-220c871c8212 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.557272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.557898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.558191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.558480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.559154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:56.559465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.560852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.568743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 43/83] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:56.568743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.590155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.593968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.594604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.594915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.595189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] PATH_INFO: `/v3/users/280e8a1dffee4eacad394696d3ff53d3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.596325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] RBAC: Authorizing `identity:get_user(user_id=280e8a1dffee4eacad394696d3ff53d3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.597805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.599348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 41/84] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/users/280e8a1dffee4eacad394696d3ff53d3 => generated 247 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:56.600201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.660534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.674503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b831cbb-29d3-4dc6-b682-ab6026023c84 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.699552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fae1d8-e3bd-4928-842b-b19bb1ac7f9b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.707648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.708289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.708576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.708860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.710037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.711697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.713382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 44/85] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:56.714211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.731980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.732592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.732878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.733243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.799012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.800398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.800944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.816426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1da7d9-d3a0-457d-9fbb-59174735f1e1 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:56.849590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:56.850199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:56.850489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:56.850765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:56.851417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:56.853451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:56.859054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:56.880250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 45/86] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:03:56 2022] POST /identity/v3/users => generated 289 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:03:56.881325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:56.909236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 42/87] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:03:56.909963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:57.011801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a361e93-c9ac-48d0-ae01-37dfa9aadcb6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:57.025310 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6774f3ea-c26a-4e2a-a161-311b81970423 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:57.055534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:57.056400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:57.056836 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:57.057156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:57.058348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:57.059869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:57.061693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 46/88] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:57.062250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:57.064843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:57.065492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:57.065775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:57.066050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:57.073631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:57.102981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:57.155154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:57.156827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:57.197030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714b6fde-ff26-4087-b831-5f3ac4bf10c5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:57.239943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:57.240793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:57.241254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:57.241534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:57.242810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=7ebc4b5874764e32b082febc76d09d5a) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:57.243124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:57.244585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:57.252501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 47/89] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:03:57 2022] GET /identity/v3/role_assignments?user.id=7ebc4b5874764e32b082febc76d09d5a&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=d261fc9497d44fedb05d9b45875771be => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:57.252501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:57.282144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 43/90] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:03:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:03:57.283236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:57.343396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:57.381305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b059382c-277d-4736-81eb-b546ed6fb632 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:57.383296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:57.457266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:57.458315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:57.458804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:57.459290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:57.460717 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:57.463081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:57.465385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 48/91] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:03:57 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:57.466402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:57.467219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1afbd0-0b1f-4209-a1cd-eb6302f60a62 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:57.531059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:57.532059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:57.532554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:57.533056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:57.612034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b456bd-106e-4492-9e1c-a0219223d36e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:57.660976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:57.661623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:57.661904 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:57.662179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:57.663835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:57.666549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:57.668176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 49/92] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:03:57 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:57.668947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:57.669958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:57.671417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:57.697209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:57.785287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:57.864425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cf6cf0-49da-4bca-96b6-a8a0219bedf8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:57.892961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 44/93] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:03:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 598 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:03:57.892961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:57.919010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:57.919636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:57.919926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:57.920208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:57.921444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] RBAC: Adding query filter params (role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=7ebc4b5874764e32b082febc76d09d5a) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:57.921788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:57.923256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:57.934394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 50/94] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:03:57 2022] GET /identity/v3/role_assignments?user.id=7ebc4b5874764e32b082febc76d09d5a&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 259 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:57.935265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:58.062654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:58.123270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72010a48-d8c2-490a-8539-2c07b9f7685a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:58.200249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:58.200917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:58.201229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:58.201510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] PATH_INFO: `/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:58.205654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] RBAC: Authorizing `identity:get_project(project_id=10d35d2d8d67457eb6c4e45f105c6e8c)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:58.207157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:58.208889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 51/95] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:03:57 2022] GET /identity/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c => generated 312 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:58.217269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:03:58.224567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814e0a77-e5d7-4ea7-b1cd-14317a6ec4f9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:58.335270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49114af-9ec2-4348-accf-9cda08e10beb None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:03:58.338665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e777ba-1ab2-4304-9aa6-2d50fb085f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:58.344202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e777ba-1ab2-4304-9aa6-2d50fb085f99 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:58.344507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e777ba-1ab2-4304-9aa6-2d50fb085f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:58.345039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e777ba-1ab2-4304-9aa6-2d50fb085f99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:58.346415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e777ba-1ab2-4304-9aa6-2d50fb085f99 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=280e8a1dffee4eacad394696d3ff53d3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:58.347542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e777ba-1ab2-4304-9aa6-2d50fb085f99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:58.349485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e777ba-1ab2-4304-9aa6-2d50fb085f99 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:58.363406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 45/96] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:03:57 2022] GET /identity/v3/role_assignments?user.id=280e8a1dffee4eacad394696d3ff53d3&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 482 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:58.389036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c302906c-eea3-456c-824e-45753e5972c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:03:58.389604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c302906c-eea3-456c-824e-45753e5972c7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:58.389895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c302906c-eea3-456c-824e-45753e5972c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:58.390175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c302906c-eea3-456c-824e-45753e5972c7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:58.390930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c302906c-eea3-456c-824e-45753e5972c7 None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=10d35d2d8d67457eb6c4e45f105c6e8c, user.id=7ebc4b5874764e32b082febc76d09d5a) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:03:58.391245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c302906c-eea3-456c-824e-45753e5972c7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:03:58.392754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c302906c-eea3-456c-824e-45753e5972c7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:03:58.406088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 52/97] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:03:58 2022] GET /identity/v3/role_assignments?user.id=7ebc4b5874764e32b082febc76d09d5a&scope.project.id=10d35d2d8d67457eb6c4e45f105c6e8c&role.id=d261fc9497d44fedb05d9b45875771be => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:03:59.747211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed36f7b-52e0-4e2c-95ab-f365042b90a7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:59.747741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed36f7b-52e0-4e2c-95ab-f365042b90a7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:59.747816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed36f7b-52e0-4e2c-95ab-f365042b90a7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:59.749157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 46/98] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:03:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:03:59.754564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad269d73-90f9-4800-87e0-f9aa7e186087 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:03:59.754985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad269d73-90f9-4800-87e0-f9aa7e186087 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:03:59.755159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad269d73-90f9-4800-87e0-f9aa7e186087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:03:59.763564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:03:59.776240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ad269d73-90f9-4800-87e0-f9aa7e186087 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:03:59.777419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ad269d73-90f9-4800-87e0-f9aa7e186087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:03:59.863354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 53/99] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:03:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:00.091134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e9a7d8-60b7-43bb-9aa5-09692f66021c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:00.091483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e9a7d8-60b7-43bb-9aa5-09692f66021c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:00.091650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e9a7d8-60b7-43bb-9aa5-09692f66021c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:00.093389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 47/100] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:00.098809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777489ea-84cd-4da6-b45b-14851f88aec1 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:00.099103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777489ea-84cd-4da6-b45b-14851f88aec1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:00.099382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777489ea-84cd-4da6-b45b-14851f88aec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:00.132538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-777489ea-84cd-4da6-b45b-14851f88aec1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:00.134664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-777489ea-84cd-4da6-b45b-14851f88aec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:00.189847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42612b9b-2965-4143-bf26-24b4d0e8385d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:00.190140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42612b9b-2965-4143-bf26-24b4d0e8385d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:00.190407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42612b9b-2965-4143-bf26-24b4d0e8385d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:00.191863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 48/101] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:00.197651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa17cf0-c6ff-45b1-b7ea-1b7e338f1984 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:00.197944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa17cf0-c6ff-45b1-b7ea-1b7e338f1984 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:00.198207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa17cf0-c6ff-45b1-b7ea-1b7e338f1984 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:00.222343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3aa17cf0-c6ff-45b1-b7ea-1b7e338f1984 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:00.223542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3aa17cf0-c6ff-45b1-b7ea-1b7e338f1984 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:00.243626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 54/102] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:00.259309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:00.316588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 49/103] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:00.440605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777489ea-84cd-4da6-b45b-14851f88aec1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:00.538565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:00.587895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777489ea-84cd-4da6-b45b-14851f88aec1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:00.632512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:00.634555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:00.635077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:00.635581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:00.645999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:00.651876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:00.652722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:00.653505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:00.655681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 55/104] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:00.662457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa17cf0-c6ff-45b1-b7ea-1b7e338f1984 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:00.690488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:00.846076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c6f49-3e88-4b69-b5ad-7cd78faac97f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:00.853974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa17cf0-c6ff-45b1-b7ea-1b7e338f1984 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:00.913579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:00.924004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:00.924644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:00.924956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:00.925229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:00.926615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:00.928624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:00.930739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:00.931310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 56/105] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:00.931722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:00.932014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:00.932129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:00.932282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:00.932985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:00.933293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:00.934661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:00.952304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 50/106] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:00.953201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:00.987488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:01.093702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af6f162-b34a-4bbe-b3e7-d11261e968ed None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:01.131456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852781fb-7d54-485d-9ceb-50872ef99532 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:01.134958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:01.135578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.135863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.136141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.136796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:01.139252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:01.185567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 51/107] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:04:00 2022] POST /identity/v3/projects => generated 302 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:01.187284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:01.250818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:01.251480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.251767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.252157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.293811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:01.299061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:01.299966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:01.301166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 57/108] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 368 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:01.309987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:01.319028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:01.381597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:01.396728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366734ab-7153-4207-8028-456bdfbeb590 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:01.456479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d878b-568e-4566-8752-7a638207d346 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:01.458958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b42582-140d-4951-b7f6-140c47b38794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:01.459724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b42582-140d-4951-b7f6-140c47b38794 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.461009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b42582-140d-4951-b7f6-140c47b38794 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.461370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b42582-140d-4951-b7f6-140c47b38794 None admin] PATH_INFO: `/v3/users/e228728753234fd88155868b2146506a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.480609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b42582-140d-4951-b7f6-140c47b38794 None admin] RBAC: Authorizing `identity:get_user(user_id=e228728753234fd88155868b2146506a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:01.483014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b42582-140d-4951-b7f6-140c47b38794 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:01.485792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 52/109] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:01 2022] GET /identity/v3/users/e228728753234fd88155868b2146506a => generated 289 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:01.487815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250580eb-ff6d-4800-bbff-8a81c44f4c4c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.488096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250580eb-ff6d-4800-bbff-8a81c44f4c4c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.488612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250580eb-ff6d-4800-bbff-8a81c44f4c4c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.490641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 53/110] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:01.491443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250580eb-ff6d-4800-bbff-8a81c44f4c4c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:01.507160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad5ad44-b05f-4bb5-ac68-524752b4f5be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:01.507160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad5ad44-b05f-4bb5-ac68-524752b4f5be None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.507160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad5ad44-b05f-4bb5-ac68-524752b4f5be None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.507160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad5ad44-b05f-4bb5-ac68-524752b4f5be None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.507548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad5ad44-b05f-4bb5-ac68-524752b4f5be None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:01.507548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad5ad44-b05f-4bb5-ac68-524752b4f5be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:01.513201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad5ad44-b05f-4bb5-ac68-524752b4f5be None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:01.526950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 58/111] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:01.532990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b1e791-e0a9-44d7-bcd0-817ca4d0e9aa None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.532990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b1e791-e0a9-44d7-bcd0-817ca4d0e9aa None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.532990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b1e791-e0a9-44d7-bcd0-817ca4d0e9aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.555748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-28b1e791-e0a9-44d7-bcd0-817ca4d0e9aa None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:01.557090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-28b1e791-e0a9-44d7-bcd0-817ca4d0e9aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:01.631849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250580eb-ff6d-4800-bbff-8a81c44f4c4c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:01.635720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 59/112] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:01.636586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b1e791-e0a9-44d7-bcd0-817ca4d0e9aa None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:01.665167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:01.676155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:01.677648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.678105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.678548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.715475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:01.763006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:01.764746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:01.822205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b1e791-e0a9-44d7-bcd0-817ca4d0e9aa None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:01.876965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 54/113] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:01.883109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:01.899363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:01.900658 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:01.901130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:01.901578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:01.904420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:01.906881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:01.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 60/114] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:01 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:01.917564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:01.956114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:02.083849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da2c85-db80-4c3b-8108-26abb5e27e2c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:02.084034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d82c47-30fe-4d6b-96dc-78636836b610 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:02.098931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:02.119539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7fc716-bf68-4d5d-ab6e-248dbd92005c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:02.120359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7fc716-bf68-4d5d-ab6e-248dbd92005c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.120649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7fc716-bf68-4d5d-ab6e-248dbd92005c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.120957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7fc716-bf68-4d5d-ab6e-248dbd92005c None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.122100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7fc716-bf68-4d5d-ab6e-248dbd92005c None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:02.123566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7fc716-bf68-4d5d-ab6e-248dbd92005c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:02.124443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:02.125394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 55/115] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:01 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:02.125667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.126094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.126343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a9e5a-1986-4de4-a45f-720a2e77193b None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.126642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a9e5a-1986-4de4-a45f-720a2e77193b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.126734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.126920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94a9e5a-1986-4de4-a45f-720a2e77193b None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.128123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 56/116] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:02.128302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:02.129164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1015457b-9c81-4cfa-91ed-59438be109cb None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.129455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1015457b-9c81-4cfa-91ed-59438be109cb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.129736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1015457b-9c81-4cfa-91ed-59438be109cb None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.130342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:02.130920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 57/117] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:02.131622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1015457b-9c81-4cfa-91ed-59438be109cb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:02.132801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 61/118] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:01 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:02.134410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:02.236787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:02.239412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a73499f-6380-41c0-a0e8-73ef1d2d7a60 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:02.246962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1015457b-9c81-4cfa-91ed-59438be109cb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:02.272892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39449a1-22e7-42e2-96e8-c2b0411b1186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:02.273579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39449a1-22e7-42e2-96e8-c2b0411b1186 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.273864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39449a1-22e7-42e2-96e8-c2b0411b1186 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.274141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39449a1-22e7-42e2-96e8-c2b0411b1186 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.274865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39449a1-22e7-42e2-96e8-c2b0411b1186 None admin] RBAC: Adding query filter params (role.id=0af4a2b511b941d080bd3a12f3fe04cf, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=e228728753234fd88155868b2146506a) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:02.275176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39449a1-22e7-42e2-96e8-c2b0411b1186 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:02.276627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39449a1-22e7-42e2-96e8-c2b0411b1186 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:02.282132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 62/119] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:02 2022] GET /identity/v3/role_assignments?user.id=e228728753234fd88155868b2146506a&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=0af4a2b511b941d080bd3a12f3fe04cf => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:02.283344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e51edfc-c219-40d6-87dc-b4042b117cd7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.283890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e51edfc-c219-40d6-87dc-b4042b117cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.284189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e51edfc-c219-40d6-87dc-b4042b117cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.295543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833e9530-9026-4dc3-bea3-e660e3cdf555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:02.296646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e9530-9026-4dc3-bea3-e660e3cdf555 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.297204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e9530-9026-4dc3-bea3-e660e3cdf555 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.297680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e9530-9026-4dc3-bea3-e660e3cdf555 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.305640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0e51edfc-c219-40d6-87dc-b4042b117cd7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:02.306943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0e51edfc-c219-40d6-87dc-b4042b117cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:02.314149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833e9530-9026-4dc3-bea3-e660e3cdf555 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-833e9530-9026-4dc3-bea3-e660e3cdf555 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:02.322769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:02.323677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:02.324443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:02.326222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 58/120] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:02.330992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb642a-ad1a-48d4-9827-a02e9bae7430 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.330992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb642a-ad1a-48d4-9827-a02e9bae7430 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.330992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eb642a-ad1a-48d4-9827-a02e9bae7430 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.349643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-77eb642a-ad1a-48d4-9827-a02e9bae7430 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:02.350952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-77eb642a-ad1a-48d4-9827-a02e9bae7430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:02.371032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:02.409852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 63/121] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:02.416948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e51edfc-c219-40d6-87dc-b4042b117cd7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:02.436648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 59/122] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:02.438004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb642a-ad1a-48d4-9827-a02e9bae7430 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:02.547456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:02.568195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eb642a-ad1a-48d4-9827-a02e9bae7430 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:02.611916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e51edfc-c219-40d6-87dc-b4042b117cd7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:02.619991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:02.620939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.621425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.621894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.622786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:02.623302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:02.624868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:02.645232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 60/123] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:02.662020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:02.664309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:02.665012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.665298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.665573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610/users/7ebc4b5874764e32b082febc76d09d5a/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.667497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] RBAC: Authorizing `identity:create_grant(project_id=f241ce1a820f4905841585aa0b706610, user_id=7ebc4b5874764e32b082febc76d09d5a, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:02.668997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:02.697290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 64/124] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:02 2022] PUT /identity/v3/projects/f241ce1a820f4905841585aa0b706610/users/7ebc4b5874764e32b082febc76d09d5a/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:02.752977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:02.810291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36761b2-4a0c-4c5e-8865-7dc3ac3384ab None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:02.872491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:02.874057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:02.874720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:02.875002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:02.875271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:02.876413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:02.878962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:02.880794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 61/125] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:02 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:02.887052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:02.914885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:02.998730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2b8507-8801-4350-9ee6-fe9be642bf28 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.043945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.044686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.045010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.045296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.045505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bc0fff-e955-40f0-9fd9-5ad6819bda2d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.057199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:03.063079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:03.063917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:03.064682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 65/126] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:03.065383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.093661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.094318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.094609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.094889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.106566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:03.111793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:03.112646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:03.113446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 62/127] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:03.114073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.171296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:03.178496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7a37c8-518f-4a9b-b111-5e31a617ebd4 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.219317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.219962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.220247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.220526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.228946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:03.229484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:03.231426 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:03.232414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 66/128] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:03 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:03.232734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.277796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fb5005-84bd-43ba-9818-80d438c4b818 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.347781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.348403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.348688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.348985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.349665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:03.349974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:03.351370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:03.373703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 63/129] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:03.374357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.436053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f700b-a581-49d0-9cde-4e61f71e96cc None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.472201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.472941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.473242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.473523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.474615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:03.474719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:03.476122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:03.493340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 67/130] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:03.493340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.516157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:03.584270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1d5c7d-46ae-4948-924e-cd7c84069bbc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.599333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d6377d-c5f5-4842-b3b8-c8829c056b3a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.640693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.641419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.641707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.641986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] PATH_INFO: `/v3/users/280e8a1dffee4eacad394696d3ff53d3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.643663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] RBAC: Authorizing `identity:get_user(user_id=280e8a1dffee4eacad394696d3ff53d3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:03.645391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:03.647145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 68/131] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:03 2022] GET /identity/v3/users/280e8a1dffee4eacad394696d3ff53d3 => generated 247 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:03.647745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.656548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.657331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.657618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.657898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610/users/7ebc4b5874764e32b082febc76d09d5a/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.661983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] RBAC: Authorizing `identity:create_grant(project_id=f241ce1a820f4905841585aa0b706610, user_id=7ebc4b5874764e32b082febc76d09d5a, role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:03.663445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:03.687362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:03.694174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 64/132] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:03 2022] PUT /identity/v3/projects/f241ce1a820f4905841585aa0b706610/users/7ebc4b5874764e32b082febc76d09d5a/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:03.697610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.832059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:03.838193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db672e65-d7df-4399-9302-babc565c11c2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:03.875975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:03.876631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:03.876919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:03.877650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:03.878673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:03.880085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:03.885200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 69/133] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:03 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:03.891075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:03.920700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f563efb1-2f39-46e2-bc4c-b73d078c1913 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:04.051166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed35cdf-ab2d-4a6c-ac98-d73912059ef9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:04.054139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:04.055405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:04.055758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:04.056056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:04.061703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:04.063189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:04.065294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 65/134] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:03 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:04.073962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:04.131780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:04.132434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:04.132789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:04.133623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] PATH_INFO: `/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:04.136714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] RBAC: Authorizing `identity:get_project(project_id=10d35d2d8d67457eb6c4e45f105c6e8c)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:04.140873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:04.142843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 70/135] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:03 2022] GET /identity/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c => generated 312 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:04.150817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:04.191572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:04.196876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:04.236905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e802180d-27e8-4721-8bc5-df42ed0b8b1c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:04.268024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ea3e3a-8409-482f-aea6-7ed5e3869fe5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:04.274124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03ea328-d512-4367-b828-fca87454e8a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:04.275172 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03ea328-d512-4367-b828-fca87454e8a8 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:04.275658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03ea328-d512-4367-b828-fca87454e8a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:04.276133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03ea328-d512-4367-b828-fca87454e8a8 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610/users/280e8a1dffee4eacad394696d3ff53d3/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:04.278407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03ea328-d512-4367-b828-fca87454e8a8 None admin] RBAC: Authorizing `identity:create_grant(project_id=f241ce1a820f4905841585aa0b706610, user_id=280e8a1dffee4eacad394696d3ff53d3, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:04.280307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03ea328-d512-4367-b828-fca87454e8a8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:04.303458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 66/136] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:04 2022] PUT /identity/v3/projects/f241ce1a820f4905841585aa0b706610/users/280e8a1dffee4eacad394696d3ff53d3/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:04.309012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:04.309012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:04.309012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:04.309012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] PATH_INFO: `/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c/users/7ebc4b5874764e32b082febc76d09d5a/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:04.309871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] RBAC: Authorizing `identity:create_grant(project_id=10d35d2d8d67457eb6c4e45f105c6e8c, user_id=7ebc4b5874764e32b082febc76d09d5a, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:04.311154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2dcb09-9cdf-40ed-b846-65c302b80387 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:04.311614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:04.311833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2dcb09-9cdf-40ed-b846-65c302b80387 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:04.312318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2dcb09-9cdf-40ed-b846-65c302b80387 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:04.314337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 67/137] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:04 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:04.320767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe23cb-9362-4770-a5fc-e163b050fe3a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:04.321507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe23cb-9362-4770-a5fc-e163b050fe3a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:04.321993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe23cb-9362-4770-a5fc-e163b050fe3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:04.334913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 71/138] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:04 2022] PUT /identity/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c/users/7ebc4b5874764e32b082febc76d09d5a/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:04.343661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-41fe23cb-9362-4770-a5fc-e163b050fe3a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:04.345808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-41fe23cb-9362-4770-a5fc-e163b050fe3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:04.427475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 68/139] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:04.990455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:05.097815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:05.110610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872ff2e6-ed97-4c67-972c-d07740a066d3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:05.147600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b766ff6c-3404-4f74-99b6-d69df7da0cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:05.148240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b766ff6c-3404-4f74-99b6-d69df7da0cdd None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:05.148693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b766ff6c-3404-4f74-99b6-d69df7da0cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:05.149105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b766ff6c-3404-4f74-99b6-d69df7da0cdd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:05.150601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b766ff6c-3404-4f74-99b6-d69df7da0cdd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:05.152049 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b766ff6c-3404-4f74-99b6-d69df7da0cdd None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:05.153830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 72/140] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:05.159417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fe23cb-9362-4770-a5fc-e163b050fe3a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:05.241511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:05.294719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fe23cb-9362-4770-a5fc-e163b050fe3a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:05.344376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26900d9-2f93-47ae-8d36-1b5819296141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:05.345051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26900d9-2f93-47ae-8d36-1b5819296141 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:05.345347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26900d9-2f93-47ae-8d36-1b5819296141 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:05.345628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26900d9-2f93-47ae-8d36-1b5819296141 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:05.346318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26900d9-2f93-47ae-8d36-1b5819296141 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:05.347811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26900d9-2f93-47ae-8d36-1b5819296141 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:05.379762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 69/141] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:05 2022] POST /identity/v3/users => generated 283 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:05.441485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f476c24-cc79-444b-8ef8-1fd055b62c49 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:05.441809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f476c24-cc79-444b-8ef8-1fd055b62c49 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:05.442039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f476c24-cc79-444b-8ef8-1fd055b62c49 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:05.443402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 73/142] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:05.449672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482b12fd-90be-4689-88a3-b74d12683bf8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:05.449968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482b12fd-90be-4689-88a3-b74d12683bf8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:05.450252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482b12fd-90be-4689-88a3-b74d12683bf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:05.478471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-482b12fd-90be-4689-88a3-b74d12683bf8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:05.479665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-482b12fd-90be-4689-88a3-b74d12683bf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:05.578641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 70/143] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:05.817081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476b189b-c0da-43f4-9308-c316aec4255e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:05.817426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476b189b-c0da-43f4-9308-c316aec4255e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:05.817599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476b189b-c0da-43f4-9308-c316aec4255e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:05.819035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 74/144] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:05.825485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9375925d-aafc-499f-8c6e-72756e7269fb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:05.825778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9375925d-aafc-499f-8c6e-72756e7269fb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:05.826063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9375925d-aafc-499f-8c6e-72756e7269fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:05.849079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9375925d-aafc-499f-8c6e-72756e7269fb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:05.850260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9375925d-aafc-499f-8c6e-72756e7269fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:05.898999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:05.933961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 71/145] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:06.095927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476b189b-c0da-43f4-9308-c316aec4255e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:06.226032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476b189b-c0da-43f4-9308-c316aec4255e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:06.270565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:06.271211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:06.271492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:06.271763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:06.284820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:06.289842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:06.290700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:06.291477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 75/146] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:06.297044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9375925d-aafc-499f-8c6e-72756e7269fb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:06.442502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9375925d-aafc-499f-8c6e-72756e7269fb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:06.480394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:06.481052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:06.481342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:06.481622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:06.482392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:06.482608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:06.483995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:06.490942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:06.498077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 72/147] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:06.509832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:06.526455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:06.600186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:06.667257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc907621-d9fb-4610-98f4-c5be1a2e550b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:06.704154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e575b6f7-0028-4ab4-825c-e6402b6f6a4c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:06.730687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:06.731346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:06.731620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:06.731892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:06.751247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:06.756358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:06.757219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:06.758549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:06.760279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 76/148] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:06.764759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:06.769143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:06.769879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:06.770160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:06.770435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] PATH_INFO: `/v3/users/e228728753234fd88155868b2146506a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:06.772109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] RBAC: Authorizing `identity:get_user(user_id=e228728753234fd88155868b2146506a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:06.775925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:06.782345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 73/149] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:06 2022] GET /identity/v3/users/e228728753234fd88155868b2146506a => generated 289 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:06.787884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:06.937306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:06.968895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670ead24-fe34-4f34-b580-edf3c1921696 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:06.982514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff1b97-fb14-4d1e-a8df-4c9b00cc8019 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:06.988647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:07.007472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:07.008203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.008514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.009047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.009727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:07.010029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:07.011421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:07.020167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 77/150] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:07.026782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:07.068940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:07.069623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.069907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.070240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.071752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:07.073264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:07.075083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 74/151] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:06 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:07.080608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:07.150064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319ec2f5-e3b6-4556-abd7-aa191d201fdb None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:07.203033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964df3b2-6f33-4b99-a869-7d0aaf8fefcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:07.203673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964df3b2-6f33-4b99-a869-7d0aaf8fefcf None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.203955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964df3b2-6f33-4b99-a869-7d0aaf8fefcf None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.204219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964df3b2-6f33-4b99-a869-7d0aaf8fefcf None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.205959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964df3b2-6f33-4b99-a869-7d0aaf8fefcf None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:07.207401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964df3b2-6f33-4b99-a869-7d0aaf8fefcf None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:07.210351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 78/152] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:07 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:07.212312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da7a76-4703-460a-b74a-286822700464 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.212656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da7a76-4703-460a-b74a-286822700464 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.212971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37da7a76-4703-460a-b74a-286822700464 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.214173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 79/153] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:07.215671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da7a76-4703-460a-b74a-286822700464 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:07.229355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:07.302937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e07ed5-a75e-4e2b-b182-77e019bb2d97 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:07.332818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37da7a76-4703-460a-b74a-286822700464 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:07.371474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec50117-8ee7-442a-9be3-6382e6273dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:07.372323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec50117-8ee7-442a-9be3-6382e6273dc3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.372763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec50117-8ee7-442a-9be3-6382e6273dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.373224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec50117-8ee7-442a-9be3-6382e6273dc3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.406200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d3ea46-25fd-4bcd-a86c-635f1fc6723d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:07.406895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d3ea46-25fd-4bcd-a86c-635f1fc6723d None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.407177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d3ea46-25fd-4bcd-a86c-635f1fc6723d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.407451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d3ea46-25fd-4bcd-a86c-635f1fc6723d None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610/users/e228728753234fd88155868b2146506a/roles/0af4a2b511b941d080bd3a12f3fe04cf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.409692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d3ea46-25fd-4bcd-a86c-635f1fc6723d None admin] RBAC: Authorizing `identity:create_grant(project_id=f241ce1a820f4905841585aa0b706610, user_id=e228728753234fd88155868b2146506a, role_id=0af4a2b511b941d080bd3a12f3fe04cf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:07.411150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d3ea46-25fd-4bcd-a86c-635f1fc6723d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:07.435491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 75/154] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:07 2022] PUT /identity/v3/projects/f241ce1a820f4905841585aa0b706610/users/e228728753234fd88155868b2146506a/roles/0af4a2b511b941d080bd3a12f3fe04cf => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:07.436750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bd5772-73c5-462c-86f1-9e3f3d40f4d7 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.437069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bd5772-73c5-462c-86f1-9e3f3d40f4d7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.437350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bd5772-73c5-462c-86f1-9e3f3d40f4d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.493387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-50bd5772-73c5-462c-86f1-9e3f3d40f4d7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:07.494588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-50bd5772-73c5-462c-86f1-9e3f3d40f4d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:07.517254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec50117-8ee7-442a-9be3-6382e6273dc3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:07.518762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec50117-8ee7-442a-9be3-6382e6273dc3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:07.534922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:07.573318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:07.680911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 76/155] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:07.718449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 80/156] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 01 04:04:07.723772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bd5772-73c5-462c-86f1-9e3f3d40f4d7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:07.830940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799845df-05ea-4def-a321-2c227c23ed00 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.831235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799845df-05ea-4def-a321-2c227c23ed00 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.831445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799845df-05ea-4def-a321-2c227c23ed00 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.832947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 81/157] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:07.838847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1092c59-b6f9-44ae-b4e2-32ab0a4e0b4f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.839131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1092c59-b6f9-44ae-b4e2-32ab0a4e0b4f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.839405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1092c59-b6f9-44ae-b4e2-32ab0a4e0b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.855354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bd5772-73c5-462c-86f1-9e3f3d40f4d7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:07.874048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d1092c59-b6f9-44ae-b4e2-32ab0a4e0b4f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:07.876229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d1092c59-b6f9-44ae-b4e2-32ab0a4e0b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:07.910602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:07.911243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:07.911519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:07.911787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:07.914268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:07.915722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:07.917386 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 77/158] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:07 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:07.935458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:07.989309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:08.002373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 82/159] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:08.111827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bfa79d-2012-4178-8c21-78174e0073a7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:08.159189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3747041c-020c-4505-bed1-099785a29183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:08.159822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3747041c-020c-4505-bed1-099785a29183 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:08.160107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3747041c-020c-4505-bed1-099785a29183 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:08.160381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3747041c-020c-4505-bed1-099785a29183 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:08.161151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3747041c-020c-4505-bed1-099785a29183 None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=7ebc4b5874764e32b082febc76d09d5a) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:08.161466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3747041c-020c-4505-bed1-099785a29183 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:08.162922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3747041c-020c-4505-bed1-099785a29183 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:08.174114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 78/160] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:07 2022] GET /identity/v3/role_assignments?user.id=7ebc4b5874764e32b082febc76d09d5a&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=d261fc9497d44fedb05d9b45875771be => generated 608 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:08.185446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1092c59-b6f9-44ae-b4e2-32ab0a4e0b4f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:08.250345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:08.336077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1092c59-b6f9-44ae-b4e2-32ab0a4e0b4f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:08.376550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:08.377145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:08.377421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:08.377690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:08.395077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:08.400210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:08.401240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:08.402922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:08.404408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b00625-9353-4f32-a2d3-8eaf373ec5bb None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:08.405026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 83/161] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:08.405184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b00625-9353-4f32-a2d3-8eaf373ec5bb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:08.405184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b00625-9353-4f32-a2d3-8eaf373ec5bb None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:08.405184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 79/162] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:08.409471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:08.414863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b81c31-3277-47ab-b2cf-2ae9654d7bbc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:08.415192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b81c31-3277-47ab-b2cf-2ae9654d7bbc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:08.415413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b81c31-3277-47ab-b2cf-2ae9654d7bbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:08.457996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d2b81c31-3277-47ab-b2cf-2ae9654d7bbc None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:08.459221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d2b81c31-3277-47ab-b2cf-2ae9654d7bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:08.552818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:08.565414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec2c768-128d-4d3d-9366-5bd07cb17ea1 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:08.610503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 80/163] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:08.611926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b81c31-3277-47ab-b2cf-2ae9654d7bbc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:08.627530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:08.628079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:08.628361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:08.630132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:08.630132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:08.630132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:08.631013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:08.651453 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 84/164] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:08.657772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:08.678955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:08.843335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b81c31-3277-47ab-b2cf-2ae9654d7bbc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:08.859226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df1a506-135c-48fe-a026-b445bbdb0f8b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:08.916760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:08.918274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:08.918556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:08.918832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:08.919935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:08.920570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:08.920844 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:08.926650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:08.928644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:08.930253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:08.931925 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 85/165] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:08 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:08.939563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:08.947172 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:08.952109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:08.952979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:08.954364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 81/166] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:08.966180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:09.002521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:09.127858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5985bd09-be0a-465d-ac29-ef02bfc763a2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:09.134434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:09.172633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4a8159-dc60-4d97-a591-8bd53444c988 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:09.191919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:09.192537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:09.192817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:09.193117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:09.243343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:09.243969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:09.244309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:09.244586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:09.245284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:09.245592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:09.246981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:09.261732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 82/167] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:09.263778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:09.302201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:09.303702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:09.349331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:09.503692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 86/168] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 566 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:09.504460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:09.519154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8141d5c5-f20a-4f96-ba6f-671609a2497c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:09.608515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:09.609185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:09.609465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:09.609738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:09.642694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:09.649707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:09.650676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:09.651459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 83/169] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 387 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:09.652119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:09.668041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:09.765506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265b4da7-ce9a-44f6-b55a-4b3b2a03a757 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:09.827105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ce674-5a76-4029-86e7-f8c4dc365ee4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:09.845445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:09.881061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:09.882104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:09.883131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:09.883607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:09.884989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:09.886633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:09.888430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 84/170] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:09 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:09.889528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:09.893859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:09.894727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:09.895149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:09.895568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] PATH_INFO: `/v3/users/280e8a1dffee4eacad394696d3ff53d3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:09.901654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] RBAC: Authorizing `identity:get_user(user_id=280e8a1dffee4eacad394696d3ff53d3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:09.903319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:09.905255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 87/171] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:09 2022] GET /identity/v3/users/280e8a1dffee4eacad394696d3ff53d3 => generated 247 bytes in 401 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:09.906186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:09.985809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:10.057555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51def-583a-46bf-b5f0-21f1f1884a0c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:10.145670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:10.146304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.146598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.146876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.147557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:10.147870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:10.149962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:10.163383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca796d9-c9c9-4b4b-8a2a-b4eadebc8c72 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:10.164061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 85/172] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:10.164886 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:10.239268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:10.239899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.240181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.240454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.241238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] RBAC: Adding query filter params (role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=7ebc4b5874764e32b082febc76d09d5a) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:10.241552 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:10.243994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:10.260673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 88/173] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:09 2022] GET /identity/v3/role_assignments?user.id=7ebc4b5874764e32b082febc76d09d5a&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 608 bytes in 355 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:10.261696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:10.386560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:10.449112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837bdfd-88b6-4de1-831e-22975c74a2e1 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:10.467268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4425c7ec-bdd5-44b3-b1cf-fb589a971bf1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:10.514805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f260dd5-e763-48b3-be86-da1b518e3bb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:10.515456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f260dd5-e763-48b3-be86-da1b518e3bb6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.515739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f260dd5-e763-48b3-be86-da1b518e3bb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.518609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f260dd5-e763-48b3-be86-da1b518e3bb6 None admin] PATH_INFO: `/v3/users/7ebc4b5874764e32b082febc76d09d5a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.518954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f260dd5-e763-48b3-be86-da1b518e3bb6 None admin] RBAC: Authorizing `identity:get_user(user_id=7ebc4b5874764e32b082febc76d09d5a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:10.520429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f260dd5-e763-48b3-be86-da1b518e3bb6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:10.523535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 89/174] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:10 2022] GET /identity/v3/users/7ebc4b5874764e32b082febc76d09d5a => generated 275 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:10.523535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f58c30-03b2-4845-842b-2ec67ae28de5 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.523964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f58c30-03b2-4845-842b-2ec67ae28de5 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.524361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f58c30-03b2-4845-842b-2ec67ae28de5 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.525774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 90/175] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:10.526903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267c4bf-daa1-46f4-bb12-ac4408153f24 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.527185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267c4bf-daa1-46f4-bb12-ac4408153f24 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.528503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267c4bf-daa1-46f4-bb12-ac4408153f24 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.541578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9719df55-bf18-4214-b0af-8bcbeede99dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:10.542238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9719df55-bf18-4214-b0af-8bcbeede99dd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.542486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9719df55-bf18-4214-b0af-8bcbeede99dd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.542764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9719df55-bf18-4214-b0af-8bcbeede99dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.543682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 91/176] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:10 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:10.544565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267c4bf-daa1-46f4-bb12-ac4408153f24 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:10.565350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:10.650665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9719df55-bf18-4214-b0af-8bcbeede99dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:10.652626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9719df55-bf18-4214-b0af-8bcbeede99dd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:10.702627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267c4bf-daa1-46f4-bb12-ac4408153f24 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:10.740121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:10.761363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:10.763885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.764171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.764444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.792528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 86/177] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 628 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:10.794270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d5fba-4c40-489e-a23f-ae9810ba7d0d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.794877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d5fba-4c40-489e-a23f-ae9810ba7d0d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.795403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d5fba-4c40-489e-a23f-ae9810ba7d0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.815078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-832d5fba-4c40-489e-a23f-ae9810ba7d0d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:10.816639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-832d5fba-4c40-489e-a23f-ae9810ba7d0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:10.846762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:10.852304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:10.854524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:10.908441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 87/178] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:10.910197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1bee07-55e4-42ec-b442-3952711cdc22 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:10.910743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1bee07-55e4-42ec-b442-3952711cdc22 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:10.911236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1bee07-55e4-42ec-b442-3952711cdc22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:10.931253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8d1bee07-55e4-42ec-b442-3952711cdc22 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:10.932900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8d1bee07-55e4-42ec-b442-3952711cdc22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:10.977215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 92/179] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:10.978668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.034371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:11.047969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 88/180] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:11.056840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1bee07-55e4-42ec-b442-3952711cdc22 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.142704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35696431-73e9-484f-afb7-2b1cc2dce083 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.178213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb40328-dbe8-46bb-abcd-7a97ae898645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.179098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb40328-dbe8-46bb-abcd-7a97ae898645 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.179664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb40328-dbe8-46bb-abcd-7a97ae898645 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.180284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb40328-dbe8-46bb-abcd-7a97ae898645 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.181657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb40328-dbe8-46bb-abcd-7a97ae898645 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:11.183318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb40328-dbe8-46bb-abcd-7a97ae898645 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:11.185130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 93/181] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:10 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:11.186525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e7c4e3-9496-4020-bce1-756f55bdb33d None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.186999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e7c4e3-9496-4020-bce1-756f55bdb33d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.187445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e7c4e3-9496-4020-bce1-756f55bdb33d None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.188866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 94/182] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:11.190135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f7f8c-b891-45d7-aa4e-9dd38e787b0d None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.190635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f7f8c-b891-45d7-aa4e-9dd38e787b0d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.191103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f7f8c-b891-45d7-aa4e-9dd38e787b0d None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.192486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 95/183] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:11.198774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f93f4-88a5-4f74-8f4d-55fe23534f5e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.199310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f93f4-88a5-4f74-8f4d-55fe23534f5e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.199794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f93f4-88a5-4f74-8f4d-55fe23534f5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.201580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:11.210283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1bee07-55e4-42ec-b442-3952711cdc22 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.219934 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8c4f93f4-88a5-4f74-8f4d-55fe23534f5e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:11.221210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8c4f93f4-88a5-4f74-8f4d-55fe23534f5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:11.256304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.256978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.257248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.257585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] PATH_INFO: `/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.258760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] RBAC: Authorizing `identity:get_project(project_id=10d35d2d8d67457eb6c4e45f105c6e8c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:11.260199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:11.261846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 89/184] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/projects/10d35d2d8d67457eb6c4e45f105c6e8c => generated 312 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:11.262595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.306065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 96/185] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:11.307547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dfa3f3-a0a5-40d8-bf36-687d4c761603 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.307833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dfa3f3-a0a5-40d8-bf36-687d4c761603 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.308101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dfa3f3-a0a5-40d8-bf36-687d4c761603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.327348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-71dfa3f3-a0a5-40d8-bf36-687d4c761603 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:11.328770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-71dfa3f3-a0a5-40d8-bf36-687d4c761603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:11.349845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:11.392694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2706ec1d-d55b-4dc2-ad66-1908f35f7c85 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.405761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 97/186] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:11.406718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dfa3f3-a0a5-40d8-bf36-687d4c761603 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.444077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.445336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.445837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.446308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.447730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=280e8a1dffee4eacad394696d3ff53d3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:11.448242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:11.450339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:11.458725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 90/187] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/role_assignments?user.id=280e8a1dffee4eacad394696d3ff53d3&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:11.459722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.531525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:11.563821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dfa3f3-a0a5-40d8-bf36-687d4c761603 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.610511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.611290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.611576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.611894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.628191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:11.633924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:11.634869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:11.635646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 98/188] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:11.636295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.662952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:11.675737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bff2e07-40f4-4228-a79c-22223de8e5f3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.713254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.714101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.714519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.714928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.715806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=10d35d2d8d67457eb6c4e45f105c6e8c, user.id=7ebc4b5874764e32b082febc76d09d5a) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:11.716349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:11.717833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:11.725028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 91/189] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/role_assignments?user.id=7ebc4b5874764e32b082febc76d09d5a&scope.project.id=10d35d2d8d67457eb6c4e45f105c6e8c&role.id=d261fc9497d44fedb05d9b45875771be => generated 608 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:11.726074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.764794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30958b92-fe3a-4bda-86ac-77a40a3d9bce None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.809230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.810002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.810283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.810561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.824510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:11.829663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:11.830611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:11.831389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 99/190] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:11.833839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.837404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:11.848915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c400a2a-acfb-464f-b95e-5d09f7911d38 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.886000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.886921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.887203 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:11.887479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:11.888631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:11.890153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:11.891765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 92/191] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:11.897293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:11.925300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:11.955815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b3f9bc-8576-450a-a9dd-33cf1a7ca107 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:11.998507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:11.999395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:11.999881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.000357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.001075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:12.001387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.002762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.005979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4394bc5-7ca6-46ec-bbf8-7eaf55471b20 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.011962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 100/192] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:12.013568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.040353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.041024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.041304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.041576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.051236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:12.057725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:12.059068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 93/193] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:12.059876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.081494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.118337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eec939-e3b0-4191-b6da-276e39298a70 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.152386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.153128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.153478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.153759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.154421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:12.154725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.156090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.158208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.162392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92129fb-9ea9-4213-91e6-a9b309b05aa1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.165217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 101/194] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:12.165623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.196500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.197139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.197409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.197683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.198325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.199762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.220245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 94/195] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:12 2022] POST /identity/v3/users => generated 297 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:12.220245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.279045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b08d56-1d17-4968-8d3a-ed9ee9643cf7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.318517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.319153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.319432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.319705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] PATH_INFO: `/v3/users/f98c431599ed430a93dafe346586869e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.326413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.335531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] RBAC: Authorizing `identity:get_user(user_id=f98c431599ed430a93dafe346586869e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.337839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.339730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 102/196] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/users/f98c431599ed430a93dafe346586869e => generated 283 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:12.340427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.362869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c63f19-25f2-4afb-8b36-3a78cb85aa3c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.401095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.402165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.402510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.402850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.403551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:12.403861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.405267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.412957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.415220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 95/197] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:12.416113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.469034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820ac135-0a5f-44d0-89a8-abf429c25a5b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.508884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.509911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.510203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.510484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] PATH_INFO: `/v3/users/f98c431599ed430a93dafe346586869e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.511764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] RBAC: Authorizing `identity:get_user(user_id=f98c431599ed430a93dafe346586869e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.513226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.515055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 103/198] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/users/f98c431599ed430a93dafe346586869e => generated 283 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:12.518252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.562647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b2dd32-f4ed-4095-a1c0-1eeba5c915cf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.600651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.601843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.602128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.602400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.606131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.656842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56be8470-e2e9-4d8b-833e-a2e745994fdb None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.682373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.687709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.690506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.693934 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.694755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.695946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.696409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] PATH_INFO: `/v3/users/e228728753234fd88155868b2146506a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.697748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] RBAC: Authorizing `identity:get_user(user_id=e228728753234fd88155868b2146506a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.699339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.701107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 104/199] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/users/e228728753234fd88155868b2146506a => generated 289 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:12.706367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.801244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 96/200] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:12.802173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:12.813580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48914c1f-1e6e-4893-95aa-9f4b7edf48ad None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.838450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.852606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.853597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.854176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.854637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:12.918691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:12.920060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c43d462-9364-4a93-8e8c-b1e8d1509deb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:12.930661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:12.933147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:12.962461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:12.963156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:12.963438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:12.963711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:13.052075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:13.052858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 105/201] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:13.053960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:13.054063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:13.099508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:13.165890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:13.177492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511f43c3-70bb-4250-82bb-adf177ede003 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:13.180162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 97/202] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:13.184860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:13.221284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:13.222260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:13.222781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:13.223244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:13.224624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:13.226581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:13.228432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 106/203] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:13 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:13.229430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:13.322164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab38ba-edf0-423e-bba2-37eedff66ce4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:13.350171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c213b11f-1746-4f38-bf14-e9e3b77d822a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:13.364088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:13.365020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:13.365491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:13.365955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:13.367360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:13.369024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:13.370822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 98/204] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:13 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:13.371757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:13.375798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:13.392008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383eded5-de26-4536-a531-28585eb6950c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:13.392855 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383eded5-de26-4536-a531-28585eb6950c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:13.393064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383eded5-de26-4536-a531-28585eb6950c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:13.393318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383eded5-de26-4536-a531-28585eb6950c None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:13.394434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383eded5-de26-4536-a531-28585eb6950c None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:13.395860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383eded5-de26-4536-a531-28585eb6950c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:13.397653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 107/205] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:13 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:13.398576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383eded5-de26-4536-a531-28585eb6950c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:13.432957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:13.480016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61ebed5-32fb-430f-89cf-f289ba59b039 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:13.522189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383eded5-de26-4536-a531-28585eb6950c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:13.526406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:13.527297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:13.527789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:13.528248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:13.529251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=f98c431599ed430a93dafe346586869e) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:13.529751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:13.531375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:13.537499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 99/206] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:13 2022] GET /identity/v3/role_assignments?user.id=f98c431599ed430a93dafe346586869e&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:13.538507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:13.587299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e921cce-9587-4342-bdad-95b669a941c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:13.588322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e921cce-9587-4342-bdad-95b669a941c5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:13.588817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e921cce-9587-4342-bdad-95b669a941c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:13.589390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e921cce-9587-4342-bdad-95b669a941c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:13.590517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e921cce-9587-4342-bdad-95b669a941c5 None admin] RBAC: Adding query filter params (role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=f98c431599ed430a93dafe346586869e) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:13.591026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e921cce-9587-4342-bdad-95b669a941c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:13.592702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e921cce-9587-4342-bdad-95b669a941c5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:13.599926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 108/207] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:13 2022] GET /identity/v3/role_assignments?user.id=f98c431599ed430a93dafe346586869e&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:13.665177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:13.705875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0d3846-8b32-4b14-950d-c1e2ac21337b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:13.757566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a1105a-385c-4895-b3fb-56e84f901af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:13.758208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a1105a-385c-4895-b3fb-56e84f901af5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:13.758486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a1105a-385c-4895-b3fb-56e84f901af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:13.758760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a1105a-385c-4895-b3fb-56e84f901af5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:13.759505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a1105a-385c-4895-b3fb-56e84f901af5 None admin] RBAC: Adding query filter params (role.id=0af4a2b511b941d080bd3a12f3fe04cf, scope.project.id=f241ce1a820f4905841585aa0b706610, user.id=e228728753234fd88155868b2146506a) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:13.759812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a1105a-385c-4895-b3fb-56e84f901af5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:13.761322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a1105a-385c-4895-b3fb-56e84f901af5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:13.771370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 100/208] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:13 2022] GET /identity/v3/role_assignments?user.id=e228728753234fd88155868b2146506a&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=0af4a2b511b941d080bd3a12f3fe04cf => generated 608 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:14.701945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be54a9bd-79ff-489e-9381-7e408f4ee3b8 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:14.702989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be54a9bd-79ff-489e-9381-7e408f4ee3b8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:14.703475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be54a9bd-79ff-489e-9381-7e408f4ee3b8 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:14.705045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 109/209] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:14.710417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060bddd-5eca-49ba-9670-8961eccac3f5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:14.711800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060bddd-5eca-49ba-9670-8961eccac3f5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:14.712287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060bddd-5eca-49ba-9670-8961eccac3f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:14.731520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b060bddd-5eca-49ba-9670-8961eccac3f5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:14.733129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b060bddd-5eca-49ba-9670-8961eccac3f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:14.740952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0fcd94-a7eb-4761-8b3d-c9ccb08d51a4 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:14.741578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0fcd94-a7eb-4761-8b3d-c9ccb08d51a4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:14.742611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0fcd94-a7eb-4761-8b3d-c9ccb08d51a4 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:14.744280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 110/210] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:14.754741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f25f-fb38-40a5-aa5e-f53f4d323816 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:14.755168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f25f-fb38-40a5-aa5e-f53f4d323816 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:14.755588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f25f-fb38-40a5-aa5e-f53f4d323816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:14.779120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-fb22f25f-fb38-40a5-aa5e-f53f4d323816 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:14.780391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fb22f25f-fb38-40a5-aa5e-f53f4d323816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:14.812301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 101/211] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:14.860473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 111/212] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:15.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060bddd-5eca-49ba-9670-8961eccac3f5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:15.152068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:15.153772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f25f-fb38-40a5-aa5e-f53f4d323816 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:15.182138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:15.209656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060bddd-5eca-49ba-9670-8961eccac3f5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:15.256840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:15.257570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.257846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.258119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.270081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f25f-fb38-40a5-aa5e-f53f4d323816 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:15.271270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:15.277777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:15.278680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:15.279485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 102/213] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:15.286727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:15.315710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:15.316579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.317045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.317458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.318694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:15.321380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:15.323159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 112/214] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:15.328735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:15.391593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0ec85b-bd21-4340-9bf9-5954cd9906a5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:15.415677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:15.428262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:15.429043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.429900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.430245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.430990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:15.431368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:15.432868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:15.438778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:15.442440 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 103/215] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:15.448968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:15.451768 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afb34c9-4d00-44b2-8949-f81c675b861b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:15.489176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:15.489785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.490067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.490335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.490994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:15.493149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:15.529759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 113/216] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:15 2022] POST /identity/v3/users => generated 297 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:04:15.565236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab50e81-6563-42ae-ba81-3606def1b452 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:15.609627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c5be2-ae20-4946-b39c-a64419091d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:15.610531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c5be2-ae20-4946-b39c-a64419091d0e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.611018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c5be2-ae20-4946-b39c-a64419091d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.611535 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c5be2-ae20-4946-b39c-a64419091d0e None admin] PATH_INFO: `/v3/users/95f28c3f27a4434bbabddc8fd17d54d4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.627785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7c5be2-ae20-4946-b39c-a64419091d0e None admin] RBAC: Authorizing `identity:get_user(user_id=95f28c3f27a4434bbabddc8fd17d54d4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:15.629642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7c5be2-ae20-4946-b39c-a64419091d0e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:15.631999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 104/217] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:15 2022] GET /identity/v3/users/95f28c3f27a4434bbabddc8fd17d54d4 => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:15.638238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:15.753873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa973d89-f429-4b1d-a791-bb4e727863e5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:15.789728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90cfc1e-5e30-4af9-baa8-81fc28013621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:15.790349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90cfc1e-5e30-4af9-baa8-81fc28013621 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.790623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90cfc1e-5e30-4af9-baa8-81fc28013621 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.790893 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90cfc1e-5e30-4af9-baa8-81fc28013621 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.807968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:15.846047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68747a-2b6d-4b80-92c9-2881971b9d9d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.846539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68747a-2b6d-4b80-92c9-2881971b9d9d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.846612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68747a-2b6d-4b80-92c9-2881971b9d9d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.848035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 105/218] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:15.854013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0740d4-af45-4609-a963-2a2304c4b5de None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:15.854297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0740d4-af45-4609-a963-2a2304c4b5de None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:15.854624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0740d4-af45-4609-a963-2a2304c4b5de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:15.870602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90cfc1e-5e30-4af9-baa8-81fc28013621 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:15.872074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90cfc1e-5e30-4af9-baa8-81fc28013621 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:15.873661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cf0740d4-af45-4609-a963-2a2304c4b5de None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:15.874824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cf0740d4-af45-4609-a963-2a2304c4b5de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:15.921474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:15.963524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 106/219] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:15.993569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 114/220] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:15.999599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0740d4-af45-4609-a963-2a2304c4b5de None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:16.108355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0740d4-af45-4609-a963-2a2304c4b5de None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:16.118616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be690d2-4c13-4752-8837-1d99fb0f0ba0 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.118965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be690d2-4c13-4752-8837-1d99fb0f0ba0 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.119176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be690d2-4c13-4752-8837-1d99fb0f0ba0 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.120856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 115/221] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:16.130423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06789eb-be74-47e6-9b60-adf6d2cb8770 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.130840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06789eb-be74-47e6-9b60-adf6d2cb8770 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.131223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06789eb-be74-47e6-9b60-adf6d2cb8770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.152683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:16.153614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.154164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.154579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.155217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b06789eb-be74-47e6-9b60-adf6d2cb8770 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:16.155970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:16.156593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b06789eb-be74-47e6-9b60-adf6d2cb8770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:16.157733 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:16.159753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 107/222] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:15 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:16.167361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:16.210325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:16.215315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:16.245078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 116/223] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:16.270766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06789eb-be74-47e6-9b60-adf6d2cb8770 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:16.278980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b149f6d-7397-41c8-ac5f-1cfd521afd2f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:16.347639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:16.348510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.350582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.352994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.354737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=95f28c3f27a4434bbabddc8fd17d54d4) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:16.355969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:16.357600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:16.372669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 108/224] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:16 2022] GET /identity/v3/role_assignments?user.id=95f28c3f27a4434bbabddc8fd17d54d4&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=d261fc9497d44fedb05d9b45875771be => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:16.431516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06789eb-be74-47e6-9b60-adf6d2cb8770 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:16.469930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:16.470932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.471356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.471803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.483444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:16.488767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:16.489899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:16.490869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:16.493310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 117/225] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:16.502093 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:16.549449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:16.609329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:16.683691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4e015a-182c-4080-95d8-c22d6ad13b92 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:16.686562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:16.713645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b1d861-acbd-403f-9dad-8947c634e4c7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:16.723479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:16.724352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.724909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.725432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.726377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:16.726936 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:16.728492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:16.737557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 109/226] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:16.743960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:16.750854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:16.751496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.751775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.752048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.762575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:16.776137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:16.777106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:16.777894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 118/227] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:16.783429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:16.915595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3773875-a937-4cae-8f81-5a335c653c15 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:16.928260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:16.937344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db33aafe-2968-4aa8-bb8f-0576233c3af2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:16.975083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:16.976083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:16.976667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:16.977290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:16.978176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:16.978718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:16.980269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:16.989494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 119/228] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:16.995544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:17.000545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:17.002104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:17.008147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:17.008660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] PATH_INFO: `/v3/users/f98c431599ed430a93dafe346586869e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:17.010614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] RBAC: Authorizing `identity:get_user(user_id=f98c431599ed430a93dafe346586869e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:17.012741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:17.014563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 110/229] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:16 2022] GET /identity/v3/users/f98c431599ed430a93dafe346586869e => generated 283 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:17.021121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:17.060907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:17.180499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61abc033-3f3c-438f-921f-87af0eb545ec None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:17.195640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604bf12-6ebe-4b09-9826-41b08e957400 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:17.219458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:17.220426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:17.220900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:17.221395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] PATH_INFO: `/v3/users/f98c431599ed430a93dafe346586869e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:17.223174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] RBAC: Authorizing `identity:get_user(user_id=f98c431599ed430a93dafe346586869e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:17.224798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:17.227416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 120/230] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:16 2022] GET /identity/v3/users/f98c431599ed430a93dafe346586869e => generated 283 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:17.233814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:17.269879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:17.271966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:17.272497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:17.273941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:17.277659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:17.279335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:17.284560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 111/231] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:17 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:17.293788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:17.298228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:17.405980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc2773-5413-4225-8791-39216c01b520 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:17.444247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:17.445428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:17.445986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:17.446453 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:17.447778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:17.449464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:17.451581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 121/232] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:17 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:17.465364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:17.504064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1dc8c9-7680-4a30-880d-ed6c15ce207c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:17.504633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:17.546729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23499e35-ead8-4975-9bae-248292032ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:17.547460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23499e35-ead8-4975-9bae-248292032ecd None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:17.547716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23499e35-ead8-4975-9bae-248292032ecd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:17.547992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23499e35-ead8-4975-9bae-248292032ecd None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/f98c431599ed430a93dafe346586869e/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:17.550205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23499e35-ead8-4975-9bae-248292032ecd None admin] RBAC: Authorizing `identity:create_grant(project_id=5de2e9eeda0e45afa245e1fec8de8bc3, user_id=f98c431599ed430a93dafe346586869e, role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:17.551775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23499e35-ead8-4975-9bae-248292032ecd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:17.567594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 112/233] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:17 2022] PUT /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/f98c431599ed430a93dafe346586869e/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:17.602437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:17.615322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fd493e-2206-4e01-92e6-279bc64284b2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:17.664551 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b855e9b-6f21-424c-961e-a81071616149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:17.665549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b855e9b-6f21-424c-961e-a81071616149 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:17.665991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b855e9b-6f21-424c-961e-a81071616149 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:17.666406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b855e9b-6f21-424c-961e-a81071616149 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/f98c431599ed430a93dafe346586869e/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:17.674531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b855e9b-6f21-424c-961e-a81071616149 None admin] RBAC: Authorizing `identity:create_grant(project_id=5de2e9eeda0e45afa245e1fec8de8bc3, user_id=f98c431599ed430a93dafe346586869e, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:17.676222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b855e9b-6f21-424c-961e-a81071616149 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:17.703644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 122/234] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:17 2022] PUT /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/f98c431599ed430a93dafe346586869e/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:18.391497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566791b9-c2ca-4da6-8760-6922a9afa5b7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:18.392013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566791b9-c2ca-4da6-8760-6922a9afa5b7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:18.392086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566791b9-c2ca-4da6-8760-6922a9afa5b7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:18.393567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 113/235] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:18.406476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c18e3-8fb5-43e9-a134-7cf0dcb2eec8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:18.406982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c18e3-8fb5-43e9-a134-7cf0dcb2eec8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:18.407058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c18e3-8fb5-43e9-a134-7cf0dcb2eec8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:18.432905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a49c18e3-8fb5-43e9-a134-7cf0dcb2eec8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:18.434120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a49c18e3-8fb5-43e9-a134-7cf0dcb2eec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:18.530246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 123/236] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:18.822485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7458d-494a-45d5-8901-eaace770ba2d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:18.822856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7458d-494a-45d5-8901-eaace770ba2d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:18.823050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d7458d-494a-45d5-8901-eaace770ba2d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:18.824702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 114/237] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:18.829666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a57818-352f-4e94-a20f-e3cd8a5a0826 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:18.830219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a57818-352f-4e94-a20f-e3cd8a5a0826 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:18.830688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a57818-352f-4e94-a20f-e3cd8a5a0826 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:18.857077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-18a57818-352f-4e94-a20f-e3cd8a5a0826 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:18.858571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-18a57818-352f-4e94-a20f-e3cd8a5a0826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:18.953380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d7458d-494a-45d5-8901-eaace770ba2d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:18.971190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 124/238] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:18.999840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35937c4-a06c-4434-8e28-1964041ca1e1 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.000124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35937c4-a06c-4434-8e28-1964041ca1e1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.000571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35937c4-a06c-4434-8e28-1964041ca1e1 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.002073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 125/239] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:19.007269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16373298-fb6b-44ab-b780-23ef154f3249 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.007803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16373298-fb6b-44ab-b780-23ef154f3249 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.008225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16373298-fb6b-44ab-b780-23ef154f3249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.038084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-16373298-fb6b-44ab-b780-23ef154f3249 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:19.042271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-16373298-fb6b-44ab-b780-23ef154f3249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:19.059080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:19.128367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 126/240] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:19.142541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d7458d-494a-45d5-8901-eaace770ba2d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.201647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:19.202286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.202564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.202837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.217945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:19.222731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:19.223628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:19.224393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 115/241] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:19.230483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16373298-fb6b-44ab-b780-23ef154f3249 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:19.321188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:19.355078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:19.382126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16373298-fb6b-44ab-b780-23ef154f3249 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.448639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:19.449448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.449711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.449992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.450673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:19.450979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:19.452343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:19.462554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 127/242] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:19.473024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:19.493113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbca0c27-d98e-4d2e-8eb3-5fcfa81cd6be None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.497269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:19.541507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:19.542601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.542884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.543424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.544782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:19.546642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:19.548730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 116/243] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:19.549523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:19.605730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f989e95e-e77a-479c-a3f4-e9a35edb274c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.643163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:19.643834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.644118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.644511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] PATH_INFO: `/v3/users/9a6828b6f3ec490b88b8dea0744bd798` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.662139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] RBAC: Authorizing `identity:get_user(user_id=9a6828b6f3ec490b88b8dea0744bd798)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:19.664164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:19.665857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 128/244] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:19 2022] GET /identity/v3/users/9a6828b6f3ec490b88b8dea0744bd798 => generated 297 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:19.666602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:19.674494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:19.686305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20627a98-fcfa-48b7-9c29-9de0b07ae498 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.725437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:19.726285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.726615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.726898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.735837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:19.739422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:19.746975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:19.748038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:19.749057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:19.750981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 117/245] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:19.752197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:19.784576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41d7656-b9ec-4557-aced-e79deb434753 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.823603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:19.824217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.824493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.824764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.825641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:19.827588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:19.852558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 129/246] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:19 2022] POST /identity/v3/users => generated 293 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:19.853293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:19.869982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faa452a-da6d-4cc1-8b67-e6550ad0a40f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.910988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:19.912158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:19.912736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:19.913314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:19.937719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:19.973415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88296abe-cf99-47c5-be9b-d8249053039b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:19.999464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:20.001798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:20.002816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:20.032079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:20.034367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.036528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.037092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.040156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:20.040700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:20.042357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:20.059915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 130/247] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:20.065752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:20.172888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 118/248] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:20.181982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:20.213946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63d292e-5549-47d1-977a-28287dd13313 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:20.243445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:20.257941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:20.258818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.259154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.259430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] PATH_INFO: `/v3/users/95f28c3f27a4434bbabddc8fd17d54d4` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.260838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] RBAC: Authorizing `identity:get_user(user_id=95f28c3f27a4434bbabddc8fd17d54d4)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:20.263048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:20.264676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 131/249] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:20 2022] GET /identity/v3/users/95f28c3f27a4434bbabddc8fd17d54d4 => generated 297 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:20.270140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:20.316504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fed623-b7a9-4574-9f98-58a43c81b3f6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:20.345322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:20.359910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00da04cb-7928-46b6-91a0-502265cfd9f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:20.360907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da04cb-7928-46b6-91a0-502265cfd9f4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.361459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da04cb-7928-46b6-91a0-502265cfd9f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.361929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da04cb-7928-46b6-91a0-502265cfd9f4 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.363321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00da04cb-7928-46b6-91a0-502265cfd9f4 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:20.365050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00da04cb-7928-46b6-91a0-502265cfd9f4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:20.367074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 119/250] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:20 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:20.368779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105ab611-ad32-48f4-971d-8f0c3d99b3f9 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.369294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105ab611-ad32-48f4-971d-8f0c3d99b3f9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.369762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105ab611-ad32-48f4-971d-8f0c3d99b3f9 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.371263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 120/251] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:20.376660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105ab611-ad32-48f4-971d-8f0c3d99b3f9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:20.417958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720a0caa-c4ef-4b46-8b40-b54336d96a80 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:20.465030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcde1fea-1678-4e71-b4b4-3d9febeea34e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:20.465738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcde1fea-1678-4e71-b4b4-3d9febeea34e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.466010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcde1fea-1678-4e71-b4b4-3d9febeea34e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.466290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcde1fea-1678-4e71-b4b4-3d9febeea34e None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.467427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcde1fea-1678-4e71-b4b4-3d9febeea34e None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:20.468876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcde1fea-1678-4e71-b4b4-3d9febeea34e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:20.470718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 132/252] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:20 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:20.472502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c703a68-d601-4bb0-94ca-ffd2e131bc98 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.472790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c703a68-d601-4bb0-94ca-ffd2e131bc98 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.473094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c703a68-d601-4bb0-94ca-ffd2e131bc98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.495981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7c703a68-d601-4bb0-94ca-ffd2e131bc98 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:20.497232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7c703a68-d601-4bb0-94ca-ffd2e131bc98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:20.509016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105ab611-ad32-48f4-971d-8f0c3d99b3f9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:20.535347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:20.554833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bae8f3-47c3-4076-984f-8cfe3118b002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:20.555756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bae8f3-47c3-4076-984f-8cfe3118b002 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.556170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bae8f3-47c3-4076-984f-8cfe3118b002 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.557894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bae8f3-47c3-4076-984f-8cfe3118b002 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.558851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bae8f3-47c3-4076-984f-8cfe3118b002 None admin] RBAC: Adding query filter params (role.id=0af4a2b511b941d080bd3a12f3fe04cf, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=9a6828b6f3ec490b88b8dea0744bd798) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:20.559371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bae8f3-47c3-4076-984f-8cfe3118b002 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:20.560962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bae8f3-47c3-4076-984f-8cfe3118b002 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:20.567127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 121/253] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:20 2022] GET /identity/v3/role_assignments?user.id=9a6828b6f3ec490b88b8dea0744bd798&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=0af4a2b511b941d080bd3a12f3fe04cf => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:20.568480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84d19a4-3754-4e55-bc7a-c44ece714aa7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.568902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84d19a4-3754-4e55-bc7a-c44ece714aa7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.569341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84d19a4-3754-4e55-bc7a-c44ece714aa7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.571178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 122/254] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:20.572080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84d19a4-3754-4e55-bc7a-c44ece714aa7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:20.592058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 133/255] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:20.593640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d517cd-ec6f-421a-bb7b-af4dca63ae12 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.593981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d517cd-ec6f-421a-bb7b-af4dca63ae12 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.594341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d517cd-ec6f-421a-bb7b-af4dca63ae12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.628321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-70d517cd-ec6f-421a-bb7b-af4dca63ae12 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:20.629571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-70d517cd-ec6f-421a-bb7b-af4dca63ae12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:20.723301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:20.750955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 134/256] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:20.752970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84d19a4-3754-4e55-bc7a-c44ece714aa7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:20.807460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:20.808605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:20.809157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:20.809627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/95f28c3f27a4434bbabddc8fd17d54d4/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:20.818126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] RBAC: Authorizing `identity:create_grant(project_id=5de2e9eeda0e45afa245e1fec8de8bc3, user_id=95f28c3f27a4434bbabddc8fd17d54d4, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:20.819627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:20.843375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 123/257] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:20 2022] PUT /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/95f28c3f27a4434bbabddc8fd17d54d4/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:20.977681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d517cd-ec6f-421a-bb7b-af4dca63ae12 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:21.103432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d517cd-ec6f-421a-bb7b-af4dca63ae12 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:21.138933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:21.139570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:21.139863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:21.140121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:21.151569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:21.156592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:21.157522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:21.158304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:21.159409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 135/258] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:21.165537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:21.223047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:21.225799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:21.301815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:21.356516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0bd9ff-3ff4-4229-b624-02b518286233 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:21.370713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95a79f-ded0-481c-8c4f-30f924442a97 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:21.395871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:21.396824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:21.397399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:21.398437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:21.399411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:21.399908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:21.401495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:21.411398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 124/259] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:21.411558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:21.412195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:21.412533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:21.412811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:21.417914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:21.426582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:21.433363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:21.434269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:21.435137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 136/260] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:21.447051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:21.599217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18820fb7-dde0-416d-be72-4a9fd9939d48 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:21.602807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:21.641476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:21.642168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:21.642449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:21.642723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] PATH_INFO: `/v3/users/f98c431599ed430a93dafe346586869e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:21.643857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] RBAC: Authorizing `identity:get_user(user_id=f98c431599ed430a93dafe346586869e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:21.646770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:21.647999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 125/261] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:21 2022] GET /identity/v3/users/f98c431599ed430a93dafe346586869e => generated 283 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:21.654957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:21.674947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:21.680977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5205d487-3e1e-4b57-b1ac-08c6e1e85c79 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:21.744379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:21.745382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:21.745854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:21.746316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:21.747189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:21.747678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:21.753369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:21.772227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 137/262] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:21.778423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:21.837146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4895fe9b-bf9f-40da-a9a1-6338b3aa8b37 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:21.875332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:21.875880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:21.876158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:21.876429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:21.941546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:21.952015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:21.956800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:22.098255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 126/263] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:22.106178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:22.113471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f580c6a1-2b4d-4ffa-b501-f5a7234d8d85 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:22.213603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:22.214610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:22.215103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:22.215563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] PATH_INFO: `/v3/users/f98c431599ed430a93dafe346586869e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:22.216885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] RBAC: Authorizing `identity:get_user(user_id=f98c431599ed430a93dafe346586869e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:22.222602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:22.224427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 138/264] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:21 2022] GET /identity/v3/users/f98c431599ed430a93dafe346586869e => generated 283 bytes in 446 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:22.229337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:22.230443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:22.297488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a4530d-9b44-4a6f-8eab-f557eefcbbe2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:22.302325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:22.390622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:22.391278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:22.391560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:22.391835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:22.396002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:22.397505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:22.399587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 127/265] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:22 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:22.405397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:22.435761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0a64b9-3ca5-48cb-8988-c421c996b57c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:22.508115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:22.509089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:22.509653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:22.510166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:22.628565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:22.630420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:22.652604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b918426b-3406-4cda-90ec-8869108765f0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:22.671471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:22.749471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 139/266] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 519 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:22.755069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:22.774314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:22.774927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:22.775205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:22.775478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:22.776219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=f98c431599ed430a93dafe346586869e) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:22.776534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:22.778060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:22.795027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 128/267] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:22 2022] GET /identity/v3/role_assignments?user.id=f98c431599ed430a93dafe346586869e&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 390 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:22.862698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e7add7-909b-461f-a661-deb38cc1d332 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:22.896200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:22.903752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badcaf75-408a-4edc-bca1-ddd7204ee2f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:22.904406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badcaf75-408a-4edc-bca1-ddd7204ee2f2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:22.904689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badcaf75-408a-4edc-bca1-ddd7204ee2f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:22.904983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badcaf75-408a-4edc-bca1-ddd7204ee2f2 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:22.909554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badcaf75-408a-4edc-bca1-ddd7204ee2f2 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:22.911169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badcaf75-408a-4edc-bca1-ddd7204ee2f2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:22.912956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 140/268] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:22 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:22.919770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:22.929589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:23.077663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf28f87-4e0a-4431-87ba-9fdd8a80c1f3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:23.124664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e673a8d-f07e-40fb-8d8e-e8f84f57235b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:23.125709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e673a8d-f07e-40fb-8d8e-e8f84f57235b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:23.126185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e673a8d-f07e-40fb-8d8e-e8f84f57235b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:23.126651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e673a8d-f07e-40fb-8d8e-e8f84f57235b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:23.127585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e673a8d-f07e-40fb-8d8e-e8f84f57235b None admin] RBAC: Adding query filter params (role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=f98c431599ed430a93dafe346586869e) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:23.128098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e673a8d-f07e-40fb-8d8e-e8f84f57235b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:23.129792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e673a8d-f07e-40fb-8d8e-e8f84f57235b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:23.141750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 129/269] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:22 2022] GET /identity/v3/role_assignments?user.id=f98c431599ed430a93dafe346586869e&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:23.450940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320106c7-c977-4aea-adef-0a7f93f13c60 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:23.451634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320106c7-c977-4aea-adef-0a7f93f13c60 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:23.452133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320106c7-c977-4aea-adef-0a7f93f13c60 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:23.453723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 141/270] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:23.459543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5c270a-d368-4604-8c1c-d5bc68d30dcb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:23.460136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5c270a-d368-4604-8c1c-d5bc68d30dcb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:23.461306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5c270a-d368-4604-8c1c-d5bc68d30dcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:23.482917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ac5c270a-d368-4604-8c1c-d5bc68d30dcb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:23.484438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ac5c270a-d368-4604-8c1c-d5bc68d30dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:23.544331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314ba1b8-b782-4e40-af01-439313d41a6c None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:23.544975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314ba1b8-b782-4e40-af01-439313d41a6c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:23.545450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314ba1b8-b782-4e40-af01-439313d41a6c None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:23.547308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 142/271] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:23.552406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4544211b-f75a-43bd-b5a9-f29bbe37b54f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:23.553018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4544211b-f75a-43bd-b5a9-f29bbe37b54f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:23.553493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4544211b-f75a-43bd-b5a9-f29bbe37b54f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:23.579143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:23.586053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 130/272] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:23.587691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-4544211b-f75a-43bd-b5a9-f29bbe37b54f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:23.589255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4544211b-f75a-43bd-b5a9-f29bbe37b54f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:23.670695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 143/273] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:23.713399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38b029a-3a1b-427e-bdee-c96990d5f2ba None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:23.713873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38b029a-3a1b-427e-bdee-c96990d5f2ba None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:23.713873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38b029a-3a1b-427e-bdee-c96990d5f2ba None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:23.715208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 131/274] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:23.720862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d4f18-dec2-40a8-ba73-b76e515e04d1 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:23.721177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d4f18-dec2-40a8-ba73-b76e515e04d1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:23.721457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d4f18-dec2-40a8-ba73-b76e515e04d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:23.742900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-879d4f18-dec2-40a8-ba73-b76e515e04d1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:23.744461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-879d4f18-dec2-40a8-ba73-b76e515e04d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:23.820952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 144/275] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:23.889286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38b029a-3a1b-427e-bdee-c96990d5f2ba None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.000067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879d4f18-dec2-40a8-ba73-b76e515e04d1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.000965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38b029a-3a1b-427e-bdee-c96990d5f2ba None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.044334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.045004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.045390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.045674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.046782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.048406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:24.050451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 132/276] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:24.055804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.060937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.107815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879d4f18-dec2-40a8-ba73-b76e515e04d1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.133613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.146071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c334ae1-83e9-4e80-88d4-cca346aff390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.146710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c334ae1-83e9-4e80-88d4-cca346aff390 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.146986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c334ae1-83e9-4e80-88d4-cca346aff390 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.147253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c334ae1-83e9-4e80-88d4-cca346aff390 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.159085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c334ae1-83e9-4e80-88d4-cca346aff390 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-3c334ae1-83e9-4e80-88d4-cca346aff390 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:24.164531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:24.165483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:24.166348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 145/277] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:24.167008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dd1930-80ef-4bf0-86b8-96be1d79b127 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.167082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dd1930-80ef-4bf0-86b8-96be1d79b127 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.167374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dd1930-80ef-4bf0-86b8-96be1d79b127 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.168792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 146/278] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:24.169522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dd1930-80ef-4bf0-86b8-96be1d79b127 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.174982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94464d7-7fd1-4a33-8c0e-37ce1b9e39f2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.208710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.209350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.209628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.209895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.211099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.212577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:24.232762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 133/279] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:24 2022] POST /identity/v3/users => generated 293 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:04:24.232762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.282789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dd1930-80ef-4bf0-86b8-96be1d79b127 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.303792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.326340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff633860-1e47-4ab5-8e78-6f26d7591939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.326986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff633860-1e47-4ab5-8e78-6f26d7591939 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.327252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff633860-1e47-4ab5-8e78-6f26d7591939 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.328322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff633860-1e47-4ab5-8e78-6f26d7591939 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.340211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff633860-1e47-4ab5-8e78-6f26d7591939 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-ff633860-1e47-4ab5-8e78-6f26d7591939 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:24.345115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:24.346041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:24.346882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 147/280] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:24.347549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5847675e-758c-4616-868c-13cb340bca53 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.348169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5847675e-758c-4616-868c-13cb340bca53 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.348169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5847675e-758c-4616-868c-13cb340bca53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.368328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5847675e-758c-4616-868c-13cb340bca53 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:24.369622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5847675e-758c-4616-868c-13cb340bca53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:24.370820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01c6d84-cf25-4e8c-9213-f2b767182e85 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.396315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.411548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.412168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.412447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.412717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.413593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:24.414660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.415474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:24.428702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 134/281] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:24.428702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.449019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 148/282] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:24.457986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5847675e-758c-4616-868c-13cb340bca53 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.557468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96eab202-3a6c-4cfd-9b7f-7666bee499dd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.582812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5847675e-758c-4616-868c-13cb340bca53 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.592669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.596341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.597533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.598030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.599248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.599973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:24.600356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.601831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:24.610528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 135/283] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:24.616228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.627628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.628326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.628618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.628954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] PATH_INFO: `/v3/users/9a6828b6f3ec490b88b8dea0744bd798` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.630874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] RBAC: Authorizing `identity:get_user(user_id=9a6828b6f3ec490b88b8dea0744bd798)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.633041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:24.635230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 149/284] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/users/9a6828b6f3ec490b88b8dea0744bd798 => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:24.641512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.674045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.736624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361461d6-902f-4d94-9a54-cf08828f1d9d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.768570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7e9746-eca4-4b80-8657-d7e523690603 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.775442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.776121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.776506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.776805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] PATH_INFO: `/v3/users/af28e1b7576a4169b6fa8e30919b44b5` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.794423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] RBAC: Authorizing `identity:get_user(user_id=af28e1b7576a4169b6fa8e30919b44b5)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.795985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:24.797692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 136/285] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/users/af28e1b7576a4169b6fa8e30919b44b5 => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:24.798458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.813917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.814481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.814764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.815043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.816301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.817880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:24.820129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 150/286] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:24.825424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.875287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.911000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3780a275-723a-44ed-bb2c-1f8004680e29 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.914319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:24.954048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.954956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.955237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.955511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.956303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c23a36-7264-448f-ac9d-74ddb501ede8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:24.968897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:24.975832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:24.976829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 137/287] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:24.978041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:24.994105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:24.994847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:24.995132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:24.995609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:24.996342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, user.id=af28e1b7576a4169b6fa8e30919b44b5, scope.system=all) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:24.996656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:24.998383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:25.004189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 151/288] 10.222.0.26 () {56 vars in 1308 bytes} [Sat Oct 1 04:04:24 2022] GET /identity/v3/role_assignments?user.id=af28e1b7576a4169b6fa8e30919b44b5&scope.system=all&role.id=d261fc9497d44fedb05d9b45875771be => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:25.005125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:25.199256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f82b94-da99-4161-a09d-110053121e91 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:25.200183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe3520d-54a6-4be6-a1a9-54ee8e14367c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:25.209000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:25.244338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:25.244973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:25.245261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:25.245529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:25.246207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:25.246505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:25.247680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:25.247896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:25.248353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:25.248638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:25.248911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/9a6828b6f3ec490b88b8dea0744bd798/roles/0af4a2b511b941d080bd3a12f3fe04cf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:25.251426 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] RBAC: Authorizing `identity:create_grant(project_id=5de2e9eeda0e45afa245e1fec8de8bc3, user_id=9a6828b6f3ec490b88b8dea0744bd798, role_id=0af4a2b511b941d080bd3a12f3fe04cf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:25.252863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:25.256752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 152/289] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:25.263002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:25.278195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 138/290] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:04:24 2022] PUT /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/users/9a6828b6f3ec490b88b8dea0744bd798/roles/0af4a2b511b941d080bd3a12f3fe04cf => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:25.405961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dbf0d-8f7d-4644-9290-89dcd2944fe0 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:25.446115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:25.446763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:25.447053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:25.447331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] PATH_INFO: `/v3/users/95f28c3f27a4434bbabddc8fd17d54d4` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:25.448488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] RBAC: Authorizing `identity:get_user(user_id=95f28c3f27a4434bbabddc8fd17d54d4)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:25.451845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:25.453406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 153/291] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:25 2022] GET /identity/v3/users/95f28c3f27a4434bbabddc8fd17d54d4 => generated 297 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:25.462132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:25.497367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:25.584807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d6c3fe-f8f4-4399-a0c5-74a7df633a77 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:25.621783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:25.622402 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:25.622681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:25.622955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:25.699136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:25.700684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:25.730794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:25.810900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 139/292] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:25.813842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:25.850594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:25.920437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de38cf4-60cf-499a-9e30-75cede02cada None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:25.955775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80e42d7-af46-4321-8567-53495289a991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:25.956416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80e42d7-af46-4321-8567-53495289a991 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:25.956702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80e42d7-af46-4321-8567-53495289a991 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:25.956995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80e42d7-af46-4321-8567-53495289a991 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:25.958130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80e42d7-af46-4321-8567-53495289a991 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:25.959567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80e42d7-af46-4321-8567-53495289a991 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:25.962097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 154/293] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:25 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:25.967899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:26.073145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e61f7b-0d28-47e9-95bc-d620c09da420 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:26.098517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:26.109971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d2f936-3507-4a95-9c32-be1459c62101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:26.111063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d2f936-3507-4a95-9c32-be1459c62101 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:26.111763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d2f936-3507-4a95-9c32-be1459c62101 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:26.112053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d2f936-3507-4a95-9c32-be1459c62101 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:26.113283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d2f936-3507-4a95-9c32-be1459c62101 None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=95f28c3f27a4434bbabddc8fd17d54d4) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:26.113653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d2f936-3507-4a95-9c32-be1459c62101 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:26.115102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d2f936-3507-4a95-9c32-be1459c62101 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:26.121715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 140/294] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:25 2022] GET /identity/v3/role_assignments?user.id=95f28c3f27a4434bbabddc8fd17d54d4&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=d261fc9497d44fedb05d9b45875771be => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:26.935450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5126e9-032b-4813-b2ab-aefdf03d39e4 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:26.935954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5126e9-032b-4813-b2ab-aefdf03d39e4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:26.936030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5126e9-032b-4813-b2ab-aefdf03d39e4 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:26.937621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 155/295] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:26.942754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed4cf82-8b46-4923-84c4-f131ce067cef None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:26.943158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed4cf82-8b46-4923-84c4-f131ce067cef None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:26.943325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed4cf82-8b46-4923-84c4-f131ce067cef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:26.962025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4ed4cf82-8b46-4923-84c4-f131ce067cef None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:26.963205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4ed4cf82-8b46-4923-84c4-f131ce067cef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:27.039249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 141/296] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:27.102665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b118f7-36f6-48d0-bb09-3cf558f9da48 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.102956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b118f7-36f6-48d0-bb09-3cf558f9da48 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.103463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b118f7-36f6-48d0-bb09-3cf558f9da48 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.105284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 156/297] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:27.110226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb0a4a-309b-4f8d-a7cd-9aadd209ff51 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.110756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb0a4a-309b-4f8d-a7cd-9aadd209ff51 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.111224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb0a4a-309b-4f8d-a7cd-9aadd209ff51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.130441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-97eb0a4a-309b-4f8d-a7cd-9aadd209ff51 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:27.131913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-97eb0a4a-309b-4f8d-a7cd-9aadd209ff51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:27.202220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 142/298] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:27.324201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b118f7-36f6-48d0-bb09-3cf558f9da48 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:27.428205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b118f7-36f6-48d0-bb09-3cf558f9da48 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:27.431543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:27.468099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:27.468657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.468952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.469245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.475345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eb0a4a-309b-4f8d-a7cd-9aadd209ff51 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:27.482354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:27.489112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:27.490638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:27.491949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:27.495495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 157/299] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:27.506258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:27.534711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:27.594977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eb0a4a-309b-4f8d-a7cd-9aadd209ff51 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:27.622667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14886d69-d13a-4ea8-928e-c813b70a1b37 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:27.633842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c52787b-a420-4c8b-aad5-f760e80b536a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:27.634791 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c52787b-a420-4c8b-aad5-f760e80b536a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.635292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c52787b-a420-4c8b-aad5-f760e80b536a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.635866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c52787b-a420-4c8b-aad5-f760e80b536a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.637272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c52787b-a420-4c8b-aad5-f760e80b536a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:27.638926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c52787b-a420-4c8b-aad5-f760e80b536a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:27.640882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 143/300] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:27.646471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf399a-4ae2-4fb6-bb7f-2f83d3fa3ebc None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.646763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf399a-4ae2-4fb6-bb7f-2f83d3fa3ebc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.647170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf399a-4ae2-4fb6-bb7f-2f83d3fa3ebc None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 144/301] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:27.649200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf399a-4ae2-4fb6-bb7f-2f83d3fa3ebc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:27.666298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531b9c89-e1e1-4687-b398-55c7d762c54a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:27.666911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531b9c89-e1e1-4687-b398-55c7d762c54a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.667193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531b9c89-e1e1-4687-b398-55c7d762c54a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.667555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531b9c89-e1e1-4687-b398-55c7d762c54a None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.668129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531b9c89-e1e1-4687-b398-55c7d762c54a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:27.668437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531b9c89-e1e1-4687-b398-55c7d762c54a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:27.669843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531b9c89-e1e1-4687-b398-55c7d762c54a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:27.678475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 158/302] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:27.681232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df15d08-2355-4445-9eb1-a580855d754f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.681724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df15d08-2355-4445-9eb1-a580855d754f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.682203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df15d08-2355-4445-9eb1-a580855d754f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.702217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0df15d08-2355-4445-9eb1-a580855d754f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:27.703419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0df15d08-2355-4445-9eb1-a580855d754f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:27.724344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:27.762757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf399a-4ae2-4fb6-bb7f-2f83d3fa3ebc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:27.777181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 159/303] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:27.778227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df15d08-2355-4445-9eb1-a580855d754f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:27.786469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:27.809176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:27.809920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.810552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.810896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] PATH_INFO: `/v3/groups` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.811658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:27.813953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:27.835443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 145/304] 10.222.0.26 () {60 vars in 1134 bytes} [Sat Oct 1 04:04:27 2022] POST /identity/v3/groups => generated 229 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:27.899001 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df15d08-2355-4445-9eb1-a580855d754f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:27.938424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:27.939980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:27.940475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:27.941023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] PATH_INFO: `/v3/users/00a6f2baa62f49738f249926e9564615` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:27.957194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] RBAC: Authorizing `identity:get_user(user_id=00a6f2baa62f49738f249926e9564615)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:27.959074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:27.961671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 160/305] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:27 2022] GET /identity/v3/users/00a6f2baa62f49738f249926e9564615 => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:27.970479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:28.059593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:28.082060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:28.087061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85613ad-8a2b-4e89-a62b-675097712f2b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:28.126065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4975e7dd-be63-4390-b850-81fbd281214a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:28.126792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975e7dd-be63-4390-b850-81fbd281214a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.127074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975e7dd-be63-4390-b850-81fbd281214a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.127344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975e7dd-be63-4390-b850-81fbd281214a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.128070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4975e7dd-be63-4390-b850-81fbd281214a None admin] RBAC: Adding query filter params (role.id=0af4a2b511b941d080bd3a12f3fe04cf, user.id=00a6f2baa62f49738f249926e9564615, scope.system=all) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:28.128375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4975e7dd-be63-4390-b850-81fbd281214a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:28.130085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4975e7dd-be63-4390-b850-81fbd281214a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:28.135983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 146/306] 10.222.0.26 () {56 vars in 1308 bytes} [Sat Oct 1 04:04:27 2022] GET /identity/v3/role_assignments?user.id=00a6f2baa62f49738f249926e9564615&scope.system=all&role.id=0af4a2b511b941d080bd3a12f3fe04cf => generated 226 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:28.137121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab56129-026a-43f1-b552-9ac2daf66c4a None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.137409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab56129-026a-43f1-b552-9ac2daf66c4a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.137681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab56129-026a-43f1-b552-9ac2daf66c4a None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.138992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 147/307] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:28.144651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa2eddf-db19-45fb-ab5a-43c4274b5e41 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.144989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa2eddf-db19-45fb-ab5a-43c4274b5e41 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.145234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa2eddf-db19-45fb-ab5a-43c4274b5e41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.166284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4fa2eddf-db19-45fb-ab5a-43c4274b5e41 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:28.168242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4fa2eddf-db19-45fb-ab5a-43c4274b5e41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:28.186249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9eda6-59ea-4f57-92d3-f0c43ca511e5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:28.217326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:28.232092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:28.232845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.233236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.233617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.249258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:28.257307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:28.258283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:28.259066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:28.265046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 148/308] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:28.268974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 161/309] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:28.274086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa2eddf-db19-45fb-ab5a-43c4274b5e41 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:28.381212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa2eddf-db19-45fb-ab5a-43c4274b5e41 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:28.417453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:28.418061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.418336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.418603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.419258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:28.419614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:28.420996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:28.429410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 149/310] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:28.435239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:28.514331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:28.540549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd49cad-daac-494a-8cf5-da6454666e45 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:28.566690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:28.575899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:28.576822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.576822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.577105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] PATH_INFO: `/v3/users/af28e1b7576a4169b6fa8e30919b44b5` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.578221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] RBAC: Authorizing `identity:get_user(user_id=af28e1b7576a4169b6fa8e30919b44b5)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:28.579644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:28.581596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 162/311] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:28 2022] GET /identity/v3/users/af28e1b7576a4169b6fa8e30919b44b5 => generated 293 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:28.587253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:28.601794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:28.706771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a71c97c-97dd-4521-a1a2-fd5c4c1479a9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:28.709156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc395ba-0b84-483f-a5f0-0796e4043bb8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:28.747748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:28.748789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.749247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.749644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] PATH_INFO: `/v3/system/users/af28e1b7576a4169b6fa8e30919b44b5/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.751082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=af28e1b7576a4169b6fa8e30919b44b5, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:28.752450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:28.753704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:28.754486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.754920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.755340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.768981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 163/312] 10.222.0.26 () {58 vars in 1334 bytes} [Sat Oct 1 04:04:28 2022] PUT /identity/v3/system/users/af28e1b7576a4169b6fa8e30919b44b5/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:28.770082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:28.776410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:28.777433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:28.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 150/313] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:28.785469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:28.800036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:28.918303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da110e-2158-4f2c-8971-2054ad368eba None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:28.957326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:28.957940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:28.958156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:28.958431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:28.959102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:28.959407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:28.961518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:28.970122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 164/314] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:28.975851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:29.076493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:29.088461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e5d207-1d91-4a00-99a1-17d8ce769cd4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:29.127389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:29.128074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:29.128363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:29.128640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] PATH_INFO: `/v3/users/9a6828b6f3ec490b88b8dea0744bd798` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:29.129814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] RBAC: Authorizing `identity:get_user(user_id=9a6828b6f3ec490b88b8dea0744bd798)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:29.131260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:29.133857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 151/315] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:28 2022] GET /identity/v3/users/9a6828b6f3ec490b88b8dea0744bd798 => generated 297 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:29.139325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:29.213336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:29.248407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd56058-18b1-4ddc-95b2-e8129504211c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:29.283107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:29.284319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:29.284793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:29.285288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:29.354392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:29.356178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:29.421659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:29.456827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 165/316] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:29.465099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:29.579695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3f9623-c0a4-40ef-b6f8-e28fd077c4cd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:29.615378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b01cdd6-405d-4cd1-bd59-206200c02b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:29.616017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01cdd6-405d-4cd1-bd59-206200c02b2e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:29.616843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01cdd6-405d-4cd1-bd59-206200c02b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:29.617235 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01cdd6-405d-4cd1-bd59-206200c02b2e None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:29.618578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b01cdd6-405d-4cd1-bd59-206200c02b2e None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:29.620028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b01cdd6-405d-4cd1-bd59-206200c02b2e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:29.622043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 152/317] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:29 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:29.627875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:29.730573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94f012-9ddc-42fa-9cf5-5472bead21c4 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:29.765880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7be5200-19b7-491f-a9d1-41132fd2285d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:29.766509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7be5200-19b7-491f-a9d1-41132fd2285d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:29.766795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7be5200-19b7-491f-a9d1-41132fd2285d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:29.767067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7be5200-19b7-491f-a9d1-41132fd2285d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:29.767819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7be5200-19b7-491f-a9d1-41132fd2285d None admin] RBAC: Adding query filter params (role.id=0af4a2b511b941d080bd3a12f3fe04cf, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3, user.id=9a6828b6f3ec490b88b8dea0744bd798) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:29.768129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7be5200-19b7-491f-a9d1-41132fd2285d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:29.769677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7be5200-19b7-491f-a9d1-41132fd2285d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:29.776968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 166/318] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:04:29 2022] GET /identity/v3/role_assignments?user.id=9a6828b6f3ec490b88b8dea0744bd798&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=0af4a2b511b941d080bd3a12f3fe04cf => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:30.140161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de82a02-5bce-4e0a-b501-3f15aecdacd7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:30.140652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de82a02-5bce-4e0a-b501-3f15aecdacd7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:30.140652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de82a02-5bce-4e0a-b501-3f15aecdacd7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:30.141979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 153/319] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:30.147264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608daab1-d676-47b2-ac11-bebc55c6303d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:30.148250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608daab1-d676-47b2-ac11-bebc55c6303d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:30.148447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608daab1-d676-47b2-ac11-bebc55c6303d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:30.166395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-608daab1-d676-47b2-ac11-bebc55c6303d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:30.167561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-608daab1-d676-47b2-ac11-bebc55c6303d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:30.195879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:30.238200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 167/320] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:30.388806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3d8dd-4e41-4732-8d99-ef27549616da None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:30.389187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3d8dd-4e41-4732-8d99-ef27549616da None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:30.389382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3d8dd-4e41-4732-8d99-ef27549616da None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:30.390888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 154/321] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:30.395717 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851b2e61-e55a-4d0d-8d96-14a7094f0aed None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:30.396005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851b2e61-e55a-4d0d-8d96-14a7094f0aed None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:30.396275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851b2e61-e55a-4d0d-8d96-14a7094f0aed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:30.415098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-851b2e61-e55a-4d0d-8d96-14a7094f0aed None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:30.416255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-851b2e61-e55a-4d0d-8d96-14a7094f0aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:30.482470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 168/322] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:30.508372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3d8dd-4e41-4732-8d99-ef27549616da None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:30.538824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:30.617868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3d8dd-4e41-4732-8d99-ef27549616da None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:30.651683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:30.652236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:30.652522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:30.652775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:30.653871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:30.655285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:30.657014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 155/323] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:30.662593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851b2e61-e55a-4d0d-8d96-14a7094f0aed None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:30.756161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:30.765608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851b2e61-e55a-4d0d-8d96-14a7094f0aed None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:30.770423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:30.801125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:30.801898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:30.802176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:30.802595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] PATH_INFO: `/v3/groups` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:30.803220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:30.804746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:30.827551 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 169/324] 10.222.0.26 () {60 vars in 1134 bytes} [Sat Oct 1 04:04:30 2022] POST /identity/v3/groups => generated 226 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:04:30.885126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:30.893300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4447b-14b8-45d4-8333-db0c8dd12560 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:30.931933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6056005e-0c8e-4634-988d-75722f91607c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:30.932589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056005e-0c8e-4634-988d-75722f91607c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:30.932866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056005e-0c8e-4634-988d-75722f91607c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:30.933192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056005e-0c8e-4634-988d-75722f91607c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:30.944481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6056005e-0c8e-4634-988d-75722f91607c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6056005e-0c8e-4634-988d-75722f91607c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:30.950852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:30.951812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:30.952627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:30.958652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 156/325] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:30 2022] GET /identity/v3/roles/reader => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:30.958820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:31.075054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7a7633-62c0-4f8a-a773-8ac05bf4dafc None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:31.101136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd74982-7bf2-49e4-a193-e7f51df68132 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:31.101412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd74982-7bf2-49e4-a193-e7f51df68132 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:31.101671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd74982-7bf2-49e4-a193-e7f51df68132 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:31.102991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 157/326] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:31.109359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699ee8b-fd81-4e83-9a97-de2c88ec8174 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:31.109589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699ee8b-fd81-4e83-9a97-de2c88ec8174 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:31.109842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699ee8b-fd81-4e83-9a97-de2c88ec8174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:31.113121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:31.113728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:31.114012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:31.114285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:31.114961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:31.115265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:31.117357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:31.128978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 170/327] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:30 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:31.131969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7699ee8b-fd81-4e83-9a97-de2c88ec8174 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:31.133895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7699ee8b-fd81-4e83-9a97-de2c88ec8174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:31.136511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:31.161183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:31.224171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 158/328] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:31.262363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2cc4c8-ef42-49c5-95ca-c747e854b2b7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:31.299352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:31.300048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:31.300325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:31.300596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] PATH_INFO: `/v3/users/00a6f2baa62f49738f249926e9564615` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:31.301796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] RBAC: Authorizing `identity:get_user(user_id=00a6f2baa62f49738f249926e9564615)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:31.303220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:31.305436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 171/329] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:31 2022] GET /identity/v3/users/00a6f2baa62f49738f249926e9564615 => generated 293 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:31.310722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699ee8b-fd81-4e83-9a97-de2c88ec8174 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:31.409561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:31.430097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699ee8b-fd81-4e83-9a97-de2c88ec8174 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:31.467324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:31.467999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:31.468284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:31.468563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] PATH_INFO: `/v3/system/users/00a6f2baa62f49738f249926e9564615/roles/0af4a2b511b941d080bd3a12f3fe04cf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:31.470067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=00a6f2baa62f49738f249926e9564615, role_id=0af4a2b511b941d080bd3a12f3fe04cf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:31.471469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:31.487384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 159/330] 10.222.0.26 () {58 vars in 1334 bytes} [Sat Oct 1 04:04:31 2022] PUT /identity/v3/system/users/00a6f2baa62f49738f249926e9564615/roles/0af4a2b511b941d080bd3a12f3fe04cf => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:31.508345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:31.605268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:31.630167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ca2b9-aa3c-4398-a6d3-e07c8c9282a9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:31.670102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:31.670758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:31.671039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:31.671314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:31.684094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:31.689885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:31.690826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:31.691666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 172/331] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:31.697759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:31.820729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfeb0b2-7eb2-47b6-920b-c48c225c4a9b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:31.859078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:31.859699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:31.859980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:31.860250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:31.861052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:31.861357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:31.862738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:31.875199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 160/332] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:31.880326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:32.006435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a4bb6f-1d3b-46fd-968c-53ea46270829 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:32.044460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be014b5c-ef16-49ca-879d-d5aff7e78a12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:32.045135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be014b5c-ef16-49ca-879d-d5aff7e78a12 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:32.045415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be014b5c-ef16-49ca-879d-d5aff7e78a12 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:32.045689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be014b5c-ef16-49ca-879d-d5aff7e78a12 None admin] PATH_INFO: `/v3/users/af28e1b7576a4169b6fa8e30919b44b5` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:32.046880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be014b5c-ef16-49ca-879d-d5aff7e78a12 None admin] RBAC: Authorizing `identity:get_user(user_id=af28e1b7576a4169b6fa8e30919b44b5)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:32.048335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be014b5c-ef16-49ca-879d-d5aff7e78a12 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:32.051197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 173/333] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:31 2022] GET /identity/v3/users/af28e1b7576a4169b6fa8e30919b44b5 => generated 293 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:32.057023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:32.073880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:32.184879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25f2b64-8dba-476c-b218-075d03693dc1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:32.233748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02826ab0-1746-4b68-b886-b0e2741364b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:32.234394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02826ab0-1746-4b68-b886-b0e2741364b9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:32.234651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02826ab0-1746-4b68-b886-b0e2741364b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:32.234926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02826ab0-1746-4b68-b886-b0e2741364b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:32.235659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02826ab0-1746-4b68-b886-b0e2741364b9 None admin] RBAC: Adding query filter params (role.id=d261fc9497d44fedb05d9b45875771be, user.id=af28e1b7576a4169b6fa8e30919b44b5, scope.system=all) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:32.235968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02826ab0-1746-4b68-b886-b0e2741364b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:32.237556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02826ab0-1746-4b68-b886-b0e2741364b9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:32.250062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 161/334] 10.222.0.26 () {56 vars in 1308 bytes} [Sat Oct 1 04:04:32 2022] GET /identity/v3/role_assignments?user.id=af28e1b7576a4169b6fa8e30919b44b5&scope.system=all&role.id=d261fc9497d44fedb05d9b45875771be => generated 510 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:33.570583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedb3ff2-c4eb-4c4e-8948-105a90355ca1 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:33.571120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedb3ff2-c4eb-4c4e-8948-105a90355ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:33.571120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedb3ff2-c4eb-4c4e-8948-105a90355ca1 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:33.572734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 174/335] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:33.578292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd4029c-2cba-499d-a711-61d9e8d773e5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:33.578816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd4029c-2cba-499d-a711-61d9e8d773e5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:33.578816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd4029c-2cba-499d-a711-61d9e8d773e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:33.605359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1fd4029c-2cba-499d-a711-61d9e8d773e5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:33.606561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1fd4029c-2cba-499d-a711-61d9e8d773e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:33.687777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:33.694270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 162/336] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:33.985748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedb3ff2-c4eb-4c4e-8948-105a90355ca1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:34.000503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:34.105870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedb3ff2-c4eb-4c4e-8948-105a90355ca1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:34.140379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:34.141042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:34.141317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:34.141582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:34.152648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:34.157976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:34.159115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:34.159890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 175/337] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:34.165920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd4029c-2cba-499d-a711-61d9e8d773e5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:34.294466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd4029c-2cba-499d-a711-61d9e8d773e5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:34.334192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143b463-562f-4863-bf60-d2a4bc6316cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:34.334802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143b463-562f-4863-bf60-d2a4bc6316cd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:34.335140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143b463-562f-4863-bf60-d2a4bc6316cd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:34.335403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143b463-562f-4863-bf60-d2a4bc6316cd None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:34.336071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143b463-562f-4863-bf60-d2a4bc6316cd None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:34.336371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143b463-562f-4863-bf60-d2a4bc6316cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:34.337747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143b463-562f-4863-bf60-d2a4bc6316cd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:34.348344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 163/338] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:34.354958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:34.452951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:34.495021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8bd406-7b52-4942-9c30-9f1f26ee28d5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:34.548356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:34.548951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:34.549243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:34.549527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:34.560385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002ca52c-1298-4265-85ba-8e3ee5a47ea9 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:34.560668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002ca52c-1298-4265-85ba-8e3ee5a47ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:34.560956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002ca52c-1298-4265-85ba-8e3ee5a47ea9 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:34.562474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 164/339] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:34.571341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ff099a-3933-46ab-8fe2-2eccc350df15 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:34.572037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ff099a-3933-46ab-8fe2-2eccc350df15 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:34.572313 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ff099a-3933-46ab-8fe2-2eccc350df15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:34.603225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-82ff099a-3933-46ab-8fe2-2eccc350df15 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:34.604407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-82ff099a-3933-46ab-8fe2-2eccc350df15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:34.677466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:34.682430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:34.699715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:34.719151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 165/340] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:34.751281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac59656-7d6c-4b26-9b47-a52e701b78ef None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:34.751783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac59656-7d6c-4b26-9b47-a52e701b78ef None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:34.752212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac59656-7d6c-4b26-9b47-a52e701b78ef None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:34.754129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 166/341] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:34.760317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968248e6-5d70-4cc1-9626-31159c1d0f8a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:34.760822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968248e6-5d70-4cc1-9626-31159c1d0f8a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:34.761296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968248e6-5d70-4cc1-9626-31159c1d0f8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:34.784386 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-968248e6-5d70-4cc1-9626-31159c1d0f8a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:34.786070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-968248e6-5d70-4cc1-9626-31159c1d0f8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:34.846726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:34.875362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 167/342] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:34.903586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 176/343] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 548 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:34.908322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968248e6-5d70-4cc1-9626-31159c1d0f8a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.062352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968248e6-5d70-4cc1-9626-31159c1d0f8a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:35.087874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:35.101567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:35.102126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.102401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.102671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.103792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:35.105302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:35.107209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 168/344] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:34 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:35.112436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.124229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.268254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f17c4-c0f1-4135-8b80-8308c081cb2e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:35.275345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbf4ca1-0eed-48b1-86a5-237f8402c2b4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:35.323010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:35.323649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.323929 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.324203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.330962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:35.331853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.332404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.332842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.336905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:35.338439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:35.340568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 177/345] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:35 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:35.342272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.346757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:35.354701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:35.355579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:35.356348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:35.359003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 169/346] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:35.360005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.421353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:35.448401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:35.483035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11a7823-baf5-4c50-93ea-da7eea3210bc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:35.528415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:35.529412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.529830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.530976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.531886 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=f241ce1a820f4905841585aa0b706610) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:35.532492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:35.534490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:35.541933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 170/347] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:35 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=d261fc9497d44fedb05d9b45875771be => generated 260 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:35.543388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.633254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cfe301-0313-4a58-9298-9037a41f418e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:35.726103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e4419f-b94d-4900-934d-71a9d21febfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:35.727077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4419f-b94d-4900-934d-71a9d21febfd None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.727246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4419f-b94d-4900-934d-71a9d21febfd None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.727508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4419f-b94d-4900-934d-71a9d21febfd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.750168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e4419f-b94d-4900-934d-71a9d21febfd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-38e4419f-b94d-4900-934d-71a9d21febfd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:35.756645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:35.757843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:35.758641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:35.760561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e24b1c9-368e-4d5d-b2f9-7f19dd5a374d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:35.765347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 178/348] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 424 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:35.766825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d4a1d-57d5-41fc-a9b5-48cd22f4ef04 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.767114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d4a1d-57d5-41fc-a9b5-48cd22f4ef04 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.767388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d4a1d-57d5-41fc-a9b5-48cd22f4ef04 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.768921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 179/349] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:35.769528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972d4a1d-57d5-41fc-a9b5-48cd22f4ef04 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.792025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:35.830425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f148789a-b504-4b16-a704-f330dbc4f5ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:35.831051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f148789a-b504-4b16-a704-f330dbc4f5ea None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.831330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f148789a-b504-4b16-a704-f330dbc4f5ea None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.831604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f148789a-b504-4b16-a704-f330dbc4f5ea None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.832278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f148789a-b504-4b16-a704-f330dbc4f5ea None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:35.832582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f148789a-b504-4b16-a704-f330dbc4f5ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:35.834002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f148789a-b504-4b16-a704-f330dbc4f5ea None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:35.853609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 171/350] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:35.854876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070aab95-7e51-4365-a2fe-a0c7cb5a8050 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.855160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070aab95-7e51-4365-a2fe-a0c7cb5a8050 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.855433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070aab95-7e51-4365-a2fe-a0c7cb5a8050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.888515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-070aab95-7e51-4365-a2fe-a0c7cb5a8050 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:35.890686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-070aab95-7e51-4365-a2fe-a0c7cb5a8050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:35.925940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:35.941654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972d4a1d-57d5-41fc-a9b5-48cd22f4ef04 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:35.979495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 172/351] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:35.980636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070aab95-7e51-4365-a2fe-a0c7cb5a8050 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:35.987058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6bbd9a-8423-4789-b3a4-48f2ced041f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:35.987924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6bbd9a-8423-4789-b3a4-48f2ced041f9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:35.988449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6bbd9a-8423-4789-b3a4-48f2ced041f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:35.988900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6bbd9a-8423-4789-b3a4-48f2ced041f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:35.989846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6bbd9a-8423-4789-b3a4-48f2ced041f9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:35.990317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6bbd9a-8423-4789-b3a4-48f2ced041f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:35.992294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6bbd9a-8423-4789-b3a4-48f2ced041f9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:36.005024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 180/352] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:36.006578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc73fa-4714-4cec-8738-b676dc3741d7 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.007587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc73fa-4714-4cec-8738-b676dc3741d7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.008026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddc73fa-4714-4cec-8738-b676dc3741d7 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.009493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 181/353] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:36.011431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc73fa-4714-4cec-8738-b676dc3741d7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:36.160691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070aab95-7e51-4365-a2fe-a0c7cb5a8050 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:36.177100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:36.199978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddc73fa-4714-4cec-8738-b676dc3741d7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:36.214140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8391b89-8c2f-402d-b557-f7a9846c1217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:36.214765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8391b89-8c2f-402d-b557-f7a9846c1217 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.215046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8391b89-8c2f-402d-b557-f7a9846c1217 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.215313 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8391b89-8c2f-402d-b557-f7a9846c1217 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.244364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bec7595-1bd4-49ac-838c-194931a05107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:36.244364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bec7595-1bd4-49ac-838c-194931a05107 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.244364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bec7595-1bd4-49ac-838c-194931a05107 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.244364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bec7595-1bd4-49ac-838c-194931a05107 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.281786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:36.313698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8391b89-8c2f-402d-b557-f7a9846c1217 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:36.315193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8391b89-8c2f-402d-b557-f7a9846c1217 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:36.346627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bec7595-1bd4-49ac-838c-194931a05107 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:36.348510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bec7595-1bd4-49ac-838c-194931a05107 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:36.445772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 173/354] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:36.447287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cdbf19-d89d-4c19-8da6-0b3509c99a3e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.447907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cdbf19-d89d-4c19-8da6-0b3509c99a3e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.448339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cdbf19-d89d-4c19-8da6-0b3509c99a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.464103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:36.471236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f1cdbf19-d89d-4c19-8da6-0b3509c99a3e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:36.472436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f1cdbf19-d89d-4c19-8da6-0b3509c99a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:36.488523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 182/355] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:36.489603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab0f765-d1e9-422e-8d5a-0d7a89afd549 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.489893 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab0f765-d1e9-422e-8d5a-0d7a89afd549 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.490172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab0f765-d1e9-422e-8d5a-0d7a89afd549 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.492370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 183/356] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:36.493147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab0f765-d1e9-422e-8d5a-0d7a89afd549 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:36.569779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 174/357] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:36.570817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cdbf19-d89d-4c19-8da6-0b3509c99a3e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:36.574039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:36.625556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab0f765-d1e9-422e-8d5a-0d7a89afd549 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:36.665131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b08cf8-1d1c-4d3c-90f7-2d1a43492267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:36.665772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b08cf8-1d1c-4d3c-90f7-2d1a43492267 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.666045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b08cf8-1d1c-4d3c-90f7-2d1a43492267 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.666317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b08cf8-1d1c-4d3c-90f7-2d1a43492267 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.677400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b08cf8-1d1c-4d3c-90f7-2d1a43492267 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-32b08cf8-1d1c-4d3c-90f7-2d1a43492267 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:36.683303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:36.684243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:36.685089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 184/358] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:36.692385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d2425e-92a9-437b-b15e-f6b5f77e1cf5 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.692727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d2425e-92a9-437b-b15e-f6b5f77e1cf5 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.692963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d2425e-92a9-437b-b15e-f6b5f77e1cf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.696997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cdbf19-d89d-4c19-8da6-0b3509c99a3e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:36.720740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c8d2425e-92a9-437b-b15e-f6b5f77e1cf5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:36.722091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c8d2425e-92a9-437b-b15e-f6b5f77e1cf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:36.746241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:36.747123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.747576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.748004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.749541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:36.750990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:36.752748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 175/359] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:36 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:36.753624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:36.768283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:36.818700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 185/360] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:36.820842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d2425e-92a9-437b-b15e-f6b5f77e1cf5 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:36.887341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:36.901956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9354ca14-b3eb-4002-bfff-8c879a1a0ead None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:36.965348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:36.967918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:36.967918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:36.967918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:36.969012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:36.970850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:36.972820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 176/361] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:36 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:36.973446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:36.976984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d2425e-92a9-437b-b15e-f6b5f77e1cf5 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.029111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.030035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.030912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.031365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.032239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:37.032725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:37.035837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:37.050696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 186/362] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:37.052218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:37.074004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:37.113052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413e0e41-f29c-475b-b8de-07d3e60bfc7f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.165045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.165687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.165966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.166232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.168394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:37.169906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:37.171617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 177/363] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:36 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:37.172390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:37.240157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb526472-7546-4db3-b419-7ca546c0088a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.254463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:37.295963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.295963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.295963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.296306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.299065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:37.302833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:37.302833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 187/364] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:37.302833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:37.314125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b5c688-c43d-44b4-883a-a2642031de45 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.356354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.357017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.357299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.357609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.370436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:37.378821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:37.387175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:37.388160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 01 04:04:37.389148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:37.391611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 178/365] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:37.392388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:37.440005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13617f4-be27-4c78-9100-9009dd039b50 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.486616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.487138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.487574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.488285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.548835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:37.555525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb323d-080b-4667-b336-cdb2ac68c0c0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.583222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:37.585181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:37.612723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.613607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.614032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.614449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.617490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=f241ce1a820f4905841585aa0b706610) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:37.617966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:37.619575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:37.626400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 179/366] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 260 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:37.629430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:37.752731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:37.764657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 188/367] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:37.766050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:37.835337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3468d-72ea-4c73-a1f1-d7fe34d1ef42 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.864116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:37.900602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba9c96c-fb8e-44e6-95a1-c78f1a6765e8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:37.901738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.902895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.903362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.903845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.917215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:37.923890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:37.924798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:37.925719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:37.929424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 180/368] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:37.931267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:37.951354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:37.951983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:37.952266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:37.952544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:37.953325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:37.953691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:37.955288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:37.962919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 189/369] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=d261fc9497d44fedb05d9b45875771be => generated 260 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:37.963620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.105800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab42645-48c7-4875-9435-77c46c84403f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.144762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.145325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.145604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.145876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.147081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.149167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.153957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 190/370] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:38.153957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.161560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c35e5b-6f53-4fe0-8050-9b100580cb41 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.173969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:38.225321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:38.240343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.241706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.242233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.242758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.243646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:38.244317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.246281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.258004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 181/371] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:38.259699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.300067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cebfcd5-8e8c-4539-be59-f884ea338a02 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.345031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d6dd40-4625-4481-94ce-cc0b79d40c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.346015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d6dd40-4625-4481-94ce-cc0b79d40c85 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.346504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d6dd40-4625-4481-94ce-cc0b79d40c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.346979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d6dd40-4625-4481-94ce-cc0b79d40c85 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.347844 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d6dd40-4625-4481-94ce-cc0b79d40c85 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:38.348364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d6dd40-4625-4481-94ce-cc0b79d40c85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.349979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d6dd40-4625-4481-94ce-cc0b79d40c85 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.359803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 191/372] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:38.365732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb95657-9bb9-47a7-b6ae-9eba068ad5b0 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.366557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb95657-9bb9-47a7-b6ae-9eba068ad5b0 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.367096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb95657-9bb9-47a7-b6ae-9eba068ad5b0 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.368770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 192/373] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:38 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:38.370323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb95657-9bb9-47a7-b6ae-9eba068ad5b0 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.390090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0010e-07eb-43f5-af7a-24787b7179f0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.436916 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.437575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.437847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.440260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.440260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.440994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.442856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 182/374] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:38 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:38.443684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.464867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:38.509819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb95657-9bb9-47a7-b6ae-9eba068ad5b0 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.520658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:38.558708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb36b5a6-9725-4d5a-b393-8d9e856b216d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.560843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb36b5a6-9725-4d5a-b393-8d9e856b216d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.560843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb36b5a6-9725-4d5a-b393-8d9e856b216d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.560843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb36b5a6-9725-4d5a-b393-8d9e856b216d None admin] PATH_INFO: `/v3/users/00a6f2baa62f49738f249926e9564615` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.561121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb36b5a6-9725-4d5a-b393-8d9e856b216d None admin] RBAC: Authorizing `identity:get_user(user_id=00a6f2baa62f49738f249926e9564615)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.562579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb36b5a6-9725-4d5a-b393-8d9e856b216d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.564471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 193/375] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:04:38 2022] GET /identity/v3/users/00a6f2baa62f49738f249926e9564615 => generated 293 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:38.565633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d57083-c82d-4dab-90d5-e9e017a3c3b4 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.565917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d57083-c82d-4dab-90d5-e9e017a3c3b4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.566195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d57083-c82d-4dab-90d5-e9e017a3c3b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.575241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ad320-f135-4dad-8efe-0382e5a897e4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.592493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-41d57083-c82d-4dab-90d5-e9e017a3c3b4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:38.593732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-41d57083-c82d-4dab-90d5-e9e017a3c3b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:38.613403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.613944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.614225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.614503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.669809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 194/376] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:38.670663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d57083-c82d-4dab-90d5-e9e017a3c3b4 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.688736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.690233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.743006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:38.769308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:38.784172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d57083-c82d-4dab-90d5-e9e017a3c3b4 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.798638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 183/377] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:38.799503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.830656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.831522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.831802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.832076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.832815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:38.833153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.837515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.841800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 195/378] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:38 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 260 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:38.841800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:38.921414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d1a47f-c0a0-4d9d-bedd-46e0ba9e8605 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:38.960540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f494b9-a752-4058-9638-df6643049246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:38.961570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f494b9-a752-4058-9638-df6643049246 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:38.962056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f494b9-a752-4058-9638-df6643049246 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:38.962527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f494b9-a752-4058-9638-df6643049246 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:38.963454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f494b9-a752-4058-9638-df6643049246 None admin] RBAC: Adding query filter params (role.id=0af4a2b511b941d080bd3a12f3fe04cf, user.id=00a6f2baa62f49738f249926e9564615, scope.system=all) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:38.963977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f494b9-a752-4058-9638-df6643049246 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:38.965768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f494b9-a752-4058-9638-df6643049246 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:38.986605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 184/379] 10.222.0.26 () {56 vars in 1308 bytes} [Sat Oct 1 04:04:38 2022] GET /identity/v3/role_assignments?user.id=00a6f2baa62f49738f249926e9564615&scope.system=all&role.id=0af4a2b511b941d080bd3a12f3fe04cf => generated 510 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:39.041457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f494b9-a752-4058-9638-df6643049246 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:39.069864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:39.097360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8230cf53-120e-4427-b717-f95bbe6c451e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.172554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:39.179505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.180230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.180573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.180910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] PATH_INFO: `/v3/projects/49c20dd6d2894e7799f891fe5467c7e0` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:39.182214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] RBAC: Authorizing `identity:get_project(project_id=49c20dd6d2894e7799f891fe5467c7e0)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:39.183682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:39.186782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 196/380] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:38 2022] GET /identity/v3/projects/49c20dd6d2894e7799f891fe5467c7e0 => generated 344 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:39.192467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:39.195471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f494b9-a752-4058-9638-df6643049246 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.233875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.234528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.234804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.235073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:39.247855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:39.253550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:39.254544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:39.255375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 185/381] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:39.263175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:39.352410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:39.362296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f909e0a-b5dd-4b5e-b88e-a770afdfcba3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.394015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e2238d-d521-40ba-8df1-31f15b09de0b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.412363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.413217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.413729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.417116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] PATH_INFO: `/v3/groups/f976714b1dc74a0190e40cf9af4e2adc` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:39.430004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] RBAC: Authorizing `identity:get_group(group_id=f976714b1dc74a0190e40cf9af4e2adc)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:39.432551 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:39.436362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 197/382] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:39 2022] GET /identity/v3/groups/f976714b1dc74a0190e40cf9af4e2adc => generated 229 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:39.441343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:39.449640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.450354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.450740 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.451101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:39.451867 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:39.452232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:39.455395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:39.467822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 186/383] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:39.473060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:39.528244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:39.592972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77556af7-ed26-44a2-9969-c885af7c3416 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.596524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56501dc7-fa47-4505-930e-6ba7c9951bcc None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.628414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.629076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.629380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.629627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:39.630746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:39.634030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:39.636007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 187/384] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:39 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:39.641283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:39.644227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:39.660634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.661287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.661667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.661949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:39.662859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] RBAC: Adding query filter params (group.id=f976714b1dc74a0190e40cf9af4e2adc, role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=49c20dd6d2894e7799f891fe5467c7e0) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:39.663318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:39.669304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:39.679234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 198/385] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:39 2022] GET /identity/v3/role_assignments?group.id=f976714b1dc74a0190e40cf9af4e2adc&scope.project.id=49c20dd6d2894e7799f891fe5467c7e0&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 260 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:39.759152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a2910d-8858-42a2-a9df-eb5784f6fc4b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.797932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f3249-03f3-4d22-a02f-7bed99a0d85b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.798587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f3249-03f3-4d22-a02f-7bed99a0d85b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.798863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f3249-03f3-4d22-a02f-7bed99a0d85b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.799133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f3249-03f3-4d22-a02f-7bed99a0d85b None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:39.800326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8f3249-03f3-4d22-a02f-7bed99a0d85b None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:39.801831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8f3249-03f3-4d22-a02f-7bed99a0d85b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:39.803910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 188/386] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:39 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:39.809677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:39.958896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73baa0b6-06b9-466f-888d-c780fba78340 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:39.986374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:39.996177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2039f185-37e7-4b18-bdf7-f64b9cd4967a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:39.997229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2039f185-37e7-4b18-bdf7-f64b9cd4967a None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:39.997915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2039f185-37e7-4b18-bdf7-f64b9cd4967a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:39.998191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2039f185-37e7-4b18-bdf7-f64b9cd4967a None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:40.000155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2039f185-37e7-4b18-bdf7-f64b9cd4967a None admin] RBAC: Authorizing `identity:create_grant(project_id=f241ce1a820f4905841585aa0b706610, group_id=6f4795575bcd45b09f84428b4db7ec3d, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:40.001629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2039f185-37e7-4b18-bdf7-f64b9cd4967a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:40.017478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 199/387] 10.222.0.26 () {58 vars in 1442 bytes} [Sat Oct 1 04:04:39 2022] PUT /identity/v3/projects/f241ce1a820f4905841585aa0b706610/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:40.788065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdbc442-77ea-41f3-b113-1aab58f741d9 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:40.788562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdbc442-77ea-41f3-b113-1aab58f741d9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:40.788562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdbc442-77ea-41f3-b113-1aab58f741d9 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:40.790304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 189/388] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:40.795940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38332dac-ed0b-4c83-b104-a60c6305d54a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:40.796344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38332dac-ed0b-4c83-b104-a60c6305d54a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:40.796516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38332dac-ed0b-4c83-b104-a60c6305d54a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:40.816968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-38332dac-ed0b-4c83-b104-a60c6305d54a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:40.818240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-38332dac-ed0b-4c83-b104-a60c6305d54a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:40.872901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48aa27c9-0cd0-4a50-b3c9-e5929395c07d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:40.873255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48aa27c9-0cd0-4a50-b3c9-e5929395c07d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:40.873510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48aa27c9-0cd0-4a50-b3c9-e5929395c07d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:40.875133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 190/389] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:40.883359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d96255-f24d-4519-8d65-8e0c1025019c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:40.883649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d96255-f24d-4519-8d65-8e0c1025019c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:40.883929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d96255-f24d-4519-8d65-8e0c1025019c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:40.912623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 200/390] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:40.913659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-73d96255-f24d-4519-8d65-8e0c1025019c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:40.914881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73d96255-f24d-4519-8d65-8e0c1025019c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:40.994911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:40.999369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 191/391] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:41.314194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38332dac-ed0b-4c83-b104-a60c6305d54a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:41.377887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d96255-f24d-4519-8d65-8e0c1025019c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:41.458639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38332dac-ed0b-4c83-b104-a60c6305d54a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:41.498890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:41.535042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:41.535685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:41.535968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:41.536242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:41.558130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:41.566431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:41.567288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:41.568444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 201/392] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:41.571216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d96255-f24d-4519-8d65-8e0c1025019c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:41.575661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:41.649563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:41.650229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:41.650508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:41.650782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:41.671106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:41.676321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:41.677223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:41.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:41.685031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 192/393] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:41.687148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:41.740572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4399dd-5897-49c6-9b7a-00b76d1a3979 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:41.782774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3207ea-d6ba-435c-9621-868bfe06558f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:41.783375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3207ea-d6ba-435c-9621-868bfe06558f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:41.783906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3207ea-d6ba-435c-9621-868bfe06558f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:41.784172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3207ea-d6ba-435c-9621-868bfe06558f None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:41.784827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3207ea-d6ba-435c-9621-868bfe06558f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:41.785160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3207ea-d6ba-435c-9621-868bfe06558f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:41.786524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3207ea-d6ba-435c-9621-868bfe06558f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:41.798105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 202/394] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:41.800512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da83a9-77c3-4f46-8bd2-d55861ccbcf1 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:41.801104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da83a9-77c3-4f46-8bd2-d55861ccbcf1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:41.801595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1da83a9-77c3-4f46-8bd2-d55861ccbcf1 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:41.803382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 203/395] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:41.805000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da83a9-77c3-4f46-8bd2-d55861ccbcf1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:41.810369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:41.861094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0671f39-96e3-4bbc-893d-d5ad555b22e1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:41.866359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:41.903986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d36355-75d0-48e4-8d27-42c510a5b9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:41.905145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d36355-75d0-48e4-8d27-42c510a5b9bd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:41.906401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d36355-75d0-48e4-8d27-42c510a5b9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:41.906874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d36355-75d0-48e4-8d27-42c510a5b9bd None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:41.907729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d36355-75d0-48e4-8d27-42c510a5b9bd None admin] RBAC: Adding query filter params (name=member) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:41.908234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d36355-75d0-48e4-8d27-42c510a5b9bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:41.910095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d36355-75d0-48e4-8d27-42c510a5b9bd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:41.921031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 193/396] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:41.922300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4c97c9-907d-4474-adf6-51eed702f0aa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:41.922587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4c97c9-907d-4474-adf6-51eed702f0aa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:41.922863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4c97c9-907d-4474-adf6-51eed702f0aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:41.941434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-bf4c97c9-907d-4474-adf6-51eed702f0aa None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:41.942734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bf4c97c9-907d-4474-adf6-51eed702f0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:41.942849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1da83a9-77c3-4f46-8bd2-d55861ccbcf1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:41.981838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:41.982519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:41.982800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:41.983072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:41.984253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:41.985901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:41.987722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 204/397] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:41 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:41.988585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.012662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 194/398] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:42.013560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4c97c9-907d-4474-adf6-51eed702f0aa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.068163 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:42.104571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86cd0e5-dd45-49f3-b5d9-49c1d318d8c0 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:42.114423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:42.116967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4c97c9-907d-4474-adf6-51eed702f0aa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:42.142441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:42.143072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.143350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.143625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.144955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:42.146401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:42.148720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 205/399] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:41 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:42.155862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:42.156147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.156713 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.157026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.157308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.158442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:42.160384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:42.162336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 195/400] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:42 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:42.174037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.269203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529de283-1fc3-452b-996f-e8150e8d6419 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:42.311618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:42.312255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.312538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.312812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.315562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:42.323688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:42.330396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 206/401] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:42 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:42.335492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.348347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3171b5-b8c8-4eac-9a81-d73cb8102f2a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:42.371716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:42.387174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:42.387840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.388122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.388400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.390618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] RBAC: Authorizing `identity:create_grant(project_id=f241ce1a820f4905841585aa0b706610, group_id=6f4795575bcd45b09f84428b4db7ec3d, role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:42.392151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:42.395939 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:42.415151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 196/402] 10.222.0.26 () {58 vars in 1442 bytes} [Sat Oct 1 04:04:42 2022] PUT /identity/v3/projects/f241ce1a820f4905841585aa0b706610/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:04:42.416076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.477290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afcd1e-319d-4679-8abb-a008aad75e50 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:42.547146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:42.549179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.549736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.550218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.555474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] RBAC: Authorizing `identity:create_grant(project_id=5de2e9eeda0e45afa245e1fec8de8bc3, group_id=6f4795575bcd45b09f84428b4db7ec3d, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:42.557309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:42.583077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 207/403] 10.222.0.26 () {58 vars in 1442 bytes} [Sat Oct 1 04:04:42 2022] PUT /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:42.600072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2ea8d5-7cc3-4b02-a20d-404b092a1294 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:42.643305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e49b95f-e743-44b0-a1a0-8d980a9596d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:42.644419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49b95f-e743-44b0-a1a0-8d980a9596d4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.644968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49b95f-e743-44b0-a1a0-8d980a9596d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.645454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e49b95f-e743-44b0-a1a0-8d980a9596d4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.660537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e49b95f-e743-44b0-a1a0-8d980a9596d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-7e49b95f-e743-44b0-a1a0-8d980a9596d4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:42.667710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:42.668698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:42.669927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:42.672488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 197/404] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:42.678186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.808053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac61d79-d2ed-4c5b-8330-5bd6a0046516 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:42.811261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:42.830465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9458f825-7d9a-47c1-8569-f6489c287706 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.830963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9458f825-7d9a-47c1-8569-f6489c287706 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.831218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9458f825-7d9a-47c1-8569-f6489c287706 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.832893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 198/405] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:42.838799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7771a2-2063-420e-8100-8847a95cfd18 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.839194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7771a2-2063-420e-8100-8847a95cfd18 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.839569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7771a2-2063-420e-8100-8847a95cfd18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.855018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:42.855635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:42.855922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:42.856201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:42.856875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:42.857214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:42.858597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:42.860942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1d7771a2-2063-420e-8100-8847a95cfd18 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:42.862174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1d7771a2-2063-420e-8100-8847a95cfd18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:42.881253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 208/406] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:42.883279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:42.933284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:42.985913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 199/407] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:42.998708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588493b-be07-4b1a-bc76-b88d3bfe30d6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:43.034112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0773f868-ad75-439c-8d43-80f4c53de214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:43.034743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0773f868-ad75-439c-8d43-80f4c53de214 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.035026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0773f868-ad75-439c-8d43-80f4c53de214 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.035304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0773f868-ad75-439c-8d43-80f4c53de214 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.036439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0773f868-ad75-439c-8d43-80f4c53de214 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:43.038077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0773f868-ad75-439c-8d43-80f4c53de214 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:43.039966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 209/408] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:42 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:43.045269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7771a2-2063-420e-8100-8847a95cfd18 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:43.201071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7771a2-2063-420e-8100-8847a95cfd18 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:43.228900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5730a-b9ef-4b25-88d6-abdb33a1de06 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.229217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5730a-b9ef-4b25-88d6-abdb33a1de06 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.229497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5730a-b9ef-4b25-88d6-abdb33a1de06 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.230967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 210/409] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:43.236702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187e2134-4a8d-4d7d-a07e-9400eaf2966b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.237016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187e2134-4a8d-4d7d-a07e-9400eaf2966b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.237296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187e2134-4a8d-4d7d-a07e-9400eaf2966b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.250953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:43.251597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.251877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.252152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.253778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:43.255548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:43.262850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 200/410] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:43 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:43.264010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-187e2134-4a8d-4d7d-a07e-9400eaf2966b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:43.265441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-187e2134-4a8d-4d7d-a07e-9400eaf2966b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:43.272352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:43.305357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:43.333205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:43.385072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 211/411] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:43.385955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187e2134-4a8d-4d7d-a07e-9400eaf2966b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:43.418442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5df2556-31fc-451a-a9fc-64756abd0e94 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:43.494703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:43.495714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.496162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187e2134-4a8d-4d7d-a07e-9400eaf2966b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:43.496397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.496852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.499391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] RBAC: Authorizing `identity:create_grant(project_id=5de2e9eeda0e45afa245e1fec8de8bc3, group_id=6f4795575bcd45b09f84428b4db7ec3d, role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:43.501088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:43.535536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 201/412] 10.222.0.26 () {58 vars in 1442 bytes} [Sat Oct 1 04:04:43 2022] PUT /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3/groups/6f4795575bcd45b09f84428b4db7ec3d/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:43.544215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:43.544958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.545248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.545521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.558585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:43.563615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:43.564607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:43.565460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 212/413] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:43.573747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:43.697777 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:43.712882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:43.721042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b993e55f-5069-4025-a14e-a24306501340 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:43.761407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:43.784701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:43.786610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.787322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.790262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.791227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:43.791739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:43.794346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:43.813585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 202/414] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:43.820234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:43.872360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f651c30e-9ece-4aca-84b7-90e6c9638671 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:43.915235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:43.915876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.916144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.916426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.932076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:43.936665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1198498-6a6a-4ce9-9888-a9947a95389a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:43.937157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:43.937979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:43.938823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 213/415] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:43.946629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:43.977644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:43.978609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:43.980506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:43.980967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] PATH_INFO: `/v3/groups/f976714b1dc74a0190e40cf9af4e2adc` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:43.982372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] RBAC: Authorizing `identity:get_group(group_id=f976714b1dc74a0190e40cf9af4e2adc)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:43.984084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:43.986288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 203/416] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:43 2022] GET /identity/v3/groups/f976714b1dc74a0190e40cf9af4e2adc => generated 229 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:43.991932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:44.025195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:44.102809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:44.113667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5feeb-fa91-4d55-8bfd-557e5fe4ef28 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:44.140726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7653a80-7446-4a4b-beec-475ac99b0264 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:44.156512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:44.157500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:44.158174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:44.159375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] PATH_INFO: `/v3/projects/49c20dd6d2894e7799f891fe5467c7e0` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:44.160897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] RBAC: Authorizing `identity:get_project(project_id=49c20dd6d2894e7799f891fe5467c7e0)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:44.163808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:44.165981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 204/417] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:43 2022] GET /identity/v3/projects/49c20dd6d2894e7799f891fe5467c7e0 => generated 344 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:44.173649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:44.196899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:44.197592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:44.197870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:44.198140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:44.198866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:44.199173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:44.202196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:44.221380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 214/418] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:44.227468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:44.372064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:44.378233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41517263-b1d6-4cd8-b055-dca9ab0bcf4a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:44.395042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ea9574-0d76-4c30-a04a-2816c3e59268 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:44.434541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:44.435229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:44.435511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:44.435784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] PATH_INFO: `/v3/projects/49c20dd6d2894e7799f891fe5467c7e0/groups/f976714b1dc74a0190e40cf9af4e2adc/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:44.439955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] RBAC: Authorizing `identity:create_grant(project_id=49c20dd6d2894e7799f891fe5467c7e0, group_id=f976714b1dc74a0190e40cf9af4e2adc, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:44.441642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:44.442775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:44.443535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:44.444104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:44.445061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:44.469668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 205/419] 10.222.0.26 () {58 vars in 1442 bytes} [Sat Oct 1 04:04:44 2022] PUT /identity/v3/projects/49c20dd6d2894e7799f891fe5467c7e0/groups/f976714b1dc74a0190e40cf9af4e2adc/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:04:44.480386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:44.539400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:44.540970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:44.656889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 215/420] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:44.666121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:44.779511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d34f9d-74f3-4e6e-9ba7-f8f46d9f639d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:44.816346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:44.817018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:44.817305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:44.817579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:44.818723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:44.821144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:44.823073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 206/421] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:44 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:44.828259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:44.915119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:44.980270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2939ac3-5346-41cb-81af-13f3edfe0f2c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:45.015081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b56859c-f07b-40b0-bf97-426c64f728f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:45.016079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b56859c-f07b-40b0-bf97-426c64f728f5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.016561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b56859c-f07b-40b0-bf97-426c64f728f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.017096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b56859c-f07b-40b0-bf97-426c64f728f5 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.018544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b56859c-f07b-40b0-bf97-426c64f728f5 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:45.020218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b56859c-f07b-40b0-bf97-426c64f728f5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:45.022421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 216/422] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:44 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:45.028694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:45.072542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:45.183793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bebe10a-3e0f-415b-bb95-147b006dc7b2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:45.196125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366f5fec-b9d5-42d1-b517-e2ee1fdeb2f0 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.196405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366f5fec-b9d5-42d1-b517-e2ee1fdeb2f0 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.196675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366f5fec-b9d5-42d1-b517-e2ee1fdeb2f0 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.199381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 217/423] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:45.203518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2f1f4-cba8-4553-b3fd-49b1a46840d1 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.203799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2f1f4-cba8-4553-b3fd-49b1a46840d1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.204068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2f1f4-cba8-4553-b3fd-49b1a46840d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.222814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f7d2f1f4-cba8-4553-b3fd-49b1a46840d1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:45.224094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f7d2f1f4-cba8-4553-b3fd-49b1a46840d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:45.235902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e8a141-fc3f-4398-bcd4-f4f88236c44c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:45.236516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8a141-fc3f-4398-bcd4-f4f88236c44c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.236914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8a141-fc3f-4398-bcd4-f4f88236c44c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.237516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8a141-fc3f-4398-bcd4-f4f88236c44c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.237942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e8a141-fc3f-4398-bcd4-f4f88236c44c None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=f241ce1a820f4905841585aa0b706610) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:45.238255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e8a141-fc3f-4398-bcd4-f4f88236c44c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:45.239820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e8a141-fc3f-4398-bcd4-f4f88236c44c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:45.253162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 207/424] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:45 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=d261fc9497d44fedb05d9b45875771be => generated 611 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:45.261230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed13c9-e4ca-43cb-8fba-49c90341c0fd None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.261621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed13c9-e4ca-43cb-8fba-49c90341c0fd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.261904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ed13c9-e4ca-43cb-8fba-49c90341c0fd None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.263465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 208/425] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:04:45.269661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0e6d28-e329-4fb1-ac0f-ac7dcce06f02 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.269962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0e6d28-e329-4fb1-ac0f-ac7dcce06f02 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.270211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0e6d28-e329-4fb1-ac0f-ac7dcce06f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.303166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0e0e6d28-e329-4fb1-ac0f-ac7dcce06f02 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:45.304485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e0e6d28-e329-4fb1-ac0f-ac7dcce06f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:45.352523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 218/426] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:45.415081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 209/427] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:45.690949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2f1f4-cba8-4553-b3fd-49b1a46840d1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:45.734526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0e6d28-e329-4fb1-ac0f-ac7dcce06f02 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:45.752309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:45.762773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:45.828652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2f1f4-cba8-4553-b3fd-49b1a46840d1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:45.865736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0e6d28-e329-4fb1-ac0f-ac7dcce06f02 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:45.879442 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:45.880194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.880577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.880968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.895603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:45.902456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:45.903337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:45.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 219/428] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:45.909865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:45.913608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:45.914476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:45.914756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:45.915024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:45.930504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:45.936263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:45.937194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:45.938014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:45.941165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 210/429] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:45.944266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.044540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7b60c3-2ddd-4ad1-85c2-85baf7562fc3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.046894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.050194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.083102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25a01fd-c110-4fc3-a764-689abc96365f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.089759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.090367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.090649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.090917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.092035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] RBAC: Adding query filter params (name=member) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:46.092344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.093755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.105111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 220/430] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:04:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:46.112198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.125287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.126023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.126364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.126707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.127447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:46.127878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.129384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.139019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 211/431] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:46.145776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.261704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a248c37-da5e-4726-b34c-eb5df5215786 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.282035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d914f7c-12af-4b0c-bebc-8d9e904b2799 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.304171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeba4f3c-a277-482c-9573-e99c19d53abe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.304802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba4f3c-a277-482c-9573-e99c19d53abe None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.305113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba4f3c-a277-482c-9573-e99c19d53abe None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.305390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba4f3c-a277-482c-9573-e99c19d53abe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.323854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa424988-0872-4d34-a8a8-4acc18b49114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.324606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa424988-0872-4d34-a8a8-4acc18b49114 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.325149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa424988-0872-4d34-a8a8-4acc18b49114 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.325450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa424988-0872-4d34-a8a8-4acc18b49114 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.339920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.382451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.399668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeba4f3c-a277-482c-9573-e99c19d53abe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.402104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeba4f3c-a277-482c-9573-e99c19d53abe None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.419793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa424988-0872-4d34-a8a8-4acc18b49114 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.421652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa424988-0872-4d34-a8a8-4acc18b49114 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.540167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 221/432] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:46.541454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1068174-377c-4254-83ee-f193e37aae67 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.541737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1068174-377c-4254-83ee-f193e37aae67 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.542007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1068174-377c-4254-83ee-f193e37aae67 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.543553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 222/433] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:46.545122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1068174-377c-4254-83ee-f193e37aae67 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.553964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 212/434] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 01 04:04:46.555609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662ac70-131e-4ac2-be46-787af23f5b24 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.556090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662ac70-131e-4ac2-be46-787af23f5b24 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.556553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662ac70-131e-4ac2-be46-787af23f5b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.575472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7662ac70-131e-4ac2-be46-787af23f5b24 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:46.576844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7662ac70-131e-4ac2-be46-787af23f5b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:46.611261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.615428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.644350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 213/435] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:46.645822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662ac70-131e-4ac2-be46-787af23f5b24 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.658803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1068174-377c-4254-83ee-f193e37aae67 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.701439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.702379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.702825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.703305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.705116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.707254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.710941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 223/436] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:46.716199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.782989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662ac70-131e-4ac2-be46-787af23f5b24 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.827622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.828397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.828742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.829250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] PATH_INFO: `/v3/projects/f241ce1a820f4905841585aa0b706610` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.830575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] RBAC: Authorizing `identity:get_project(project_id=f241ce1a820f4905841585aa0b706610)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.832738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.834955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 214/437] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/projects/f241ce1a820f4905841585aa0b706610 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:46.840429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.851811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcac0d0-9ab1-42ed-8235-ec484bf0f514 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.881293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.895976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.896749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.897192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.897562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.899275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.901251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.903864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 224/438] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:46.909992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:46.913656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:46.949712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb492a78-5fcd-4cd9-84c7-97ad5c7a9a65 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:46.988074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:46.988741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:46.989049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:46.989325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:46.990528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:46.992044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:46.993708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 215/439] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:46.994509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.035625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6396c2c7-51ec-4029-8b3b-984c8619f6f6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.073680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.074302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.074540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.074811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.075539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=d261fc9497d44fedb05d9b45875771be, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:47.075845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:47.077304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:47.083216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 225/440] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=d261fc9497d44fedb05d9b45875771be => generated 611 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:47.083990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.113258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9dd6d1-93bc-4d52-a850-a5c71d322b6c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.121993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:47.153369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e300f0b4-9715-441c-b263-9c40b6b6a733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.153937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e300f0b4-9715-441c-b263-9c40b6b6a733 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.154216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e300f0b4-9715-441c-b263-9c40b6b6a733 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.154486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e300f0b4-9715-441c-b263-9c40b6b6a733 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.167225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e300f0b4-9715-441c-b263-9c40b6b6a733 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:47.172325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e300f0b4-9715-441c-b263-9c40b6b6a733 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:47.172856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:47.173887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:47.174677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 216/441] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:47.175875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960a46ff-1e31-49e6-97bf-d44b9f20b6a1 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.175875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960a46ff-1e31-49e6-97bf-d44b9f20b6a1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.175875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960a46ff-1e31-49e6-97bf-d44b9f20b6a1 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.177044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 217/442] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:47 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:47.180963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960a46ff-1e31-49e6-97bf-d44b9f20b6a1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.224555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6aca0d-7db5-4c64-a6b4-c4652799f087 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.267892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4a4f65-417d-45b2-be4c-3da78adcc5bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.268499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4a4f65-417d-45b2-be4c-3da78adcc5bd None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.268775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4a4f65-417d-45b2-be4c-3da78adcc5bd None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.269071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4a4f65-417d-45b2-be4c-3da78adcc5bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.270006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4a4f65-417d-45b2-be4c-3da78adcc5bd None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=f241ce1a820f4905841585aa0b706610) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:47.270109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4a4f65-417d-45b2-be4c-3da78adcc5bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:47.271490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4a4f65-417d-45b2-be4c-3da78adcc5bd None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:47.278027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 226/443] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:47 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=f241ce1a820f4905841585aa0b706610&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 611 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:47.279221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe469648-c056-4c4e-8ca2-ec8eafcad7c8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.279467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe469648-c056-4c4e-8ca2-ec8eafcad7c8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.279736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe469648-c056-4c4e-8ca2-ec8eafcad7c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.300967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-fe469648-c056-4c4e-8ca2-ec8eafcad7c8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:47.302207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fe469648-c056-4c4e-8ca2-ec8eafcad7c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:47.302514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960a46ff-1e31-49e6-97bf-d44b9f20b6a1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.353788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.354486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.354705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.354979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.355666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:47.355979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:47.357609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:47.371326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 218/444] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:47.377734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.396105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 227/445] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:47.417395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:47.495530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29140dfb-75fd-4f1f-9e20-5adcdc2196f4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.536650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.537334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.537638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.537905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.616971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:47.618420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:47.647403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:47.672451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe469648-c056-4c4e-8ca2-ec8eafcad7c8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.728036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 219/446] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 01 04:04:47.733819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.742783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:47.794449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe469648-c056-4c4e-8ca2-ec8eafcad7c8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.835152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.835786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.836062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.836485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.847826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:47.851958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6ef796-4abb-4b91-ac3c-f8116b5c95c0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:47.855064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:47.855944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:47.856737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 228/447] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:04:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:47.860232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.883055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:47.891538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.892175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.892457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.892732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] PATH_INFO: `/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.893880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] RBAC: Authorizing `identity:get_project(project_id=5de2e9eeda0e45afa245e1fec8de8bc3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:47.895315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:47.896943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 220/448] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:47 2022] GET /identity/v3/projects/5de2e9eeda0e45afa245e1fec8de8bc3 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:47.902975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:47.959417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c3a035-5803-4bb5-a58c-55e6cac36a7e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:47.979392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:47.994792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:47.995421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:47.995698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:47.995968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:47.996627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:47.996954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:47.998300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:48.001568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c242f-85c3-40d3-bc4d-9fad181c700e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:48.006173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 229/449] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:48.012164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:48.039316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:48.039947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:48.040228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:48.040507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] PATH_INFO: `/v3/groups/6f4795575bcd45b09f84428b4db7ec3d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:48.041646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] RBAC: Authorizing `identity:get_group(group_id=6f4795575bcd45b09f84428b4db7ec3d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:48.043082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:48.044619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 221/450] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:47 2022] GET /identity/v3/groups/6f4795575bcd45b09f84428b4db7ec3d => generated 226 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:48.050429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:48.110092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:48.117668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e347a50d-07b6-4007-902f-52d592a8a003 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:48.148686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d7f551-2918-497e-b3c8-4fe6ce4a0df5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:48.150719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:48.151319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:48.151599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:48.151877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:48.182067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:48.182669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:48.182950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:48.183228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:48.183947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] RBAC: Adding query filter params (group.id=6f4795575bcd45b09f84428b4db7ec3d, role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:48.184257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:48.185699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:48.191187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 222/451] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:48 2022] GET /identity/v3/role_assignments?group.id=6f4795575bcd45b09f84428b4db7ec3d&scope.project.id=5de2e9eeda0e45afa245e1fec8de8bc3&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 611 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:48.201900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:48.222469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:48.223878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:48.332692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 230/452] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 01 04:04:48.338598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:48.447510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598aa5c5-c02d-416f-9cd6-9650771b362e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:48.473729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:48.482697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:48.483339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:48.483621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:48.483896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] PATH_INFO: `/v3/projects/49c20dd6d2894e7799f891fe5467c7e0` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:48.485059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] RBAC: Authorizing `identity:get_project(project_id=49c20dd6d2894e7799f891fe5467c7e0)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:48.487214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:48.488209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 223/453] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:04:48 2022] GET /identity/v3/projects/49c20dd6d2894e7799f891fe5467c7e0 => generated 344 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:48.494213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:48.580202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:48.598254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740deed-af0a-4194-8f42-6bcc4d6cc189 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:48.630897 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5cd911-bdec-459a-a2c3-7e4fe153362d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:48.631624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5cd911-bdec-459a-a2c3-7e4fe153362d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:48.631900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5cd911-bdec-459a-a2c3-7e4fe153362d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:48.632173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5cd911-bdec-459a-a2c3-7e4fe153362d None admin] PATH_INFO: `/v3/groups/f976714b1dc74a0190e40cf9af4e2adc` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:48.633287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5cd911-bdec-459a-a2c3-7e4fe153362d None admin] RBAC: Authorizing `identity:get_group(group_id=f976714b1dc74a0190e40cf9af4e2adc)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:48.634692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5cd911-bdec-459a-a2c3-7e4fe153362d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:48.636214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 231/454] 10.222.0.26 () {56 vars in 1180 bytes} [Sat Oct 1 04:04:48 2022] GET /identity/v3/groups/f976714b1dc74a0190e40cf9af4e2adc => generated 229 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:48.642067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:48.738043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d69103b-67cf-4177-99e4-3b62e5af0df8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:48.769731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2542af10-f7a5-4e73-b181-799a69b34268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:48.770324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2542af10-f7a5-4e73-b181-799a69b34268 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:48.770603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2542af10-f7a5-4e73-b181-799a69b34268 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:48.770874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2542af10-f7a5-4e73-b181-799a69b34268 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:48.771587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2542af10-f7a5-4e73-b181-799a69b34268 None admin] RBAC: Adding query filter params (group.id=f976714b1dc74a0190e40cf9af4e2adc, role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=49c20dd6d2894e7799f891fe5467c7e0) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:48.771928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2542af10-f7a5-4e73-b181-799a69b34268 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:48.773373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2542af10-f7a5-4e73-b181-799a69b34268 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:48.778811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 224/455] 10.222.0.26 () {56 vars in 1376 bytes} [Sat Oct 1 04:04:48 2022] GET /identity/v3/role_assignments?group.id=f976714b1dc74a0190e40cf9af4e2adc&scope.project.id=49c20dd6d2894e7799f891fe5467c7e0&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 611 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:51.343221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ef845-0017-407c-a21c-fbb80f82a790 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:51.343738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ef845-0017-407c-a21c-fbb80f82a790 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:51.343738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ef845-0017-407c-a21c-fbb80f82a790 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:51.345393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 232/456] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:51.350429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b4d2a9-52b1-4609-940b-7222e16ac5dd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:51.350828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b4d2a9-52b1-4609-940b-7222e16ac5dd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:51.351016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b4d2a9-52b1-4609-940b-7222e16ac5dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:51.377894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c7b4d2a9-52b1-4609-940b-7222e16ac5dd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:51.379092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c7b4d2a9-52b1-4609-940b-7222e16ac5dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:51.457678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:51.474078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 225/457] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:51.539616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb628f5-1a58-44a5-a79f-9a8c8f8f4daf None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:51.539912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb628f5-1a58-44a5-a79f-9a8c8f8f4daf None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:51.540174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb628f5-1a58-44a5-a79f-9a8c8f8f4daf None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:51.541788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 233/458] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:51.546836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f346805-0051-42c2-b31d-259b963ffc50 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:51.547365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f346805-0051-42c2-b31d-259b963ffc50 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:51.547836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f346805-0051-42c2-b31d-259b963ffc50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:51.570129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6f346805-0051-42c2-b31d-259b963ffc50 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:51.571654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6f346805-0051-42c2-b31d-259b963ffc50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:51.669912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 226/459] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:51.812432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f444bd-1348-4f3d-ad3a-83fdac4a607a None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:51.813091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f444bd-1348-4f3d-ad3a-83fdac4a607a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:51.813561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f444bd-1348-4f3d-ad3a-83fdac4a607a None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:51.815331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 234/460] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:51.821081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794fbc47-83f5-415e-a54e-de0b23fb0df4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:51.821594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794fbc47-83f5-415e-a54e-de0b23fb0df4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:51.822064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794fbc47-83f5-415e-a54e-de0b23fb0df4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:51.850443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-794fbc47-83f5-415e-a54e-de0b23fb0df4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:51.851996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-794fbc47-83f5-415e-a54e-de0b23fb0df4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:51.912512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f444bd-1348-4f3d-ad3a-83fdac4a607a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:51.962551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 227/461] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:04:52.072140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f444bd-1348-4f3d-ad3a-83fdac4a607a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:52.104204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794fbc47-83f5-415e-a54e-de0b23fb0df4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:52.126269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:52.128363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.128652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.128948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.157462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:52.161851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:52.194874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:52.195845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:52.196684 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:52.196684 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:52.196684 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:52.197911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 235/462] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:52.206943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:52.220161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:52.222921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:52.294314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794fbc47-83f5-415e-a54e-de0b23fb0df4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:52.330487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41951ef9-1513-49cd-a174-e2ce3c578ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:52.331132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41951ef9-1513-49cd-a174-e2ce3c578ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.331566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41951ef9-1513-49cd-a174-e2ce3c578ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.332068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41951ef9-1513-49cd-a174-e2ce3c578ae6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.343864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41951ef9-1513-49cd-a174-e2ce3c578ae6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:52.345356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41951ef9-1513-49cd-a174-e2ce3c578ae6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:52.347388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc6422c-121f-44c6-a931-9db77ff09e78 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-41951ef9-1513-49cd-a174-e2ce3c578ae6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:52.363234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:52.364580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:52.366303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:52.366303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:52.366303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:52.367709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 228/463] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:04:52.369153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b380e8-7d1f-4092-8c19-f8f0d6db3ea1 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.369432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b380e8-7d1f-4092-8c19-f8f0d6db3ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.369702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b380e8-7d1f-4092-8c19-f8f0d6db3ea1 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.371029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 229/464] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:52.373982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b380e8-7d1f-4092-8c19-f8f0d6db3ea1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:52.387800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:52.388416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.388699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.389005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.389654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:52.389956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:52.391319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:52.403896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 236/465] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:52.404620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:52.472326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:52.493739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b380e8-7d1f-4092-8c19-f8f0d6db3ea1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:52.495318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:52.517308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc6f4e0-8db7-4ddd-9175-ca01ab1937b3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:52.533111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d2a657-88a3-4e0f-be1e-31babd989c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:52.534020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d2a657-88a3-4e0f-be1e-31babd989c0b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.534301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d2a657-88a3-4e0f-be1e-31babd989c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.534580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d2a657-88a3-4e0f-be1e-31babd989c0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.546113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d2a657-88a3-4e0f-be1e-31babd989c0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:52.547596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d2a657-88a3-4e0f-be1e-31babd989c0b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:52.559170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:52.559781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.560065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.560331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.561005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:52.561316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:52.563029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-59d2a657-88a3-4e0f-be1e-31babd989c0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:52.573397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:52.574584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:52.575576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:52.575576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:52.575576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:52.575576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 230/466] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:52.576349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d232b2c-ae25-4d0b-a5b2-722d3577b6cb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.576964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d232b2c-ae25-4d0b-a5b2-722d3577b6cb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.577188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d232b2c-ae25-4d0b-a5b2-722d3577b6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.582394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 237/467] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:52.589656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:52.599793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0d232b2c-ae25-4d0b-a5b2-722d3577b6cb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:52.601566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0d232b2c-ae25-4d0b-a5b2-722d3577b6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:52.680998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 231/468] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:52.680998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d232b2c-ae25-4d0b-a5b2-722d3577b6cb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:52.718188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4bdf5-7752-43ba-909b-c36eed0db5bc None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:52.746009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:52.760389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:52.761141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.761605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.761952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.762785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:52.764338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:52.800738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 238/469] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:52 2022] POST /identity/v3/users => generated 246 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:52.800738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:52.804187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:52.845924 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d232b2c-ae25-4d0b-a5b2-722d3577b6cb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:52.921908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:52.922537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:52.922817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:52.923091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:52.923747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:52.924052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:52.925480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:52.965222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 232/470] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:52.966101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:53.003073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847d13fc-5380-40e1-a2bc-a56a255aac1d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:53.044057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:53.044662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.044961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.045237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.045886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:53.047307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:53.075260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 239/471] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:52 2022] POST /identity/v3/users => generated 248 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:53.076094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:53.106949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad7eb4-f1dc-4bc3-9a5b-7d353f99c1ac None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:53.135362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:53.152978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95df3cdb-5a6b-40c7-8791-ca2bf417dfaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:53.153612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95df3cdb-5a6b-40c7-8791-ca2bf417dfaa None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.153890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95df3cdb-5a6b-40c7-8791-ca2bf417dfaa None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.154160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95df3cdb-5a6b-40c7-8791-ca2bf417dfaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.173899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95df3cdb-5a6b-40c7-8791-ca2bf417dfaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:53.176237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95df3cdb-5a6b-40c7-8791-ca2bf417dfaa None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:53.180865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-95df3cdb-5a6b-40c7-8791-ca2bf417dfaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:53.199015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:53.199985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:53.200821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:53.200821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:53.200821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:53.200821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 233/472] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:53.207526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea3a0c4-5bb5-4d62-89ce-f0f83e3d8003 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.207837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea3a0c4-5bb5-4d62-89ce-f0f83e3d8003 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.208086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea3a0c4-5bb5-4d62-89ce-f0f83e3d8003 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.209910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 234/473] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:53 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:53.210692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea3a0c4-5bb5-4d62-89ce-f0f83e3d8003 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:53.223006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43a2ec-0212-40e5-a5e5-ce99f17c9101 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:53.269527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a704cb3a-76f3-4465-a118-dbae82195701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:53.270752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704cb3a-76f3-4465-a118-dbae82195701 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.271254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704cb3a-76f3-4465-a118-dbae82195701 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.271901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a704cb3a-76f3-4465-a118-dbae82195701 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.272959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704cb3a-76f3-4465-a118-dbae82195701 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:53.274632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a704cb3a-76f3-4465-a118-dbae82195701 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:53.305002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 240/474] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:53 2022] POST /identity/v3/users => generated 248 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:53.307313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dce4dc-90a6-45d0-84e2-002582cf30e7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.307822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dce4dc-90a6-45d0-84e2-002582cf30e7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.308334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dce4dc-90a6-45d0-84e2-002582cf30e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.335499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b9dce4dc-90a6-45d0-84e2-002582cf30e7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:53.337506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b9dce4dc-90a6-45d0-84e2-002582cf30e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:53.339232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea3a0c4-5bb5-4d62-89ce-f0f83e3d8003 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:53.384699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:53.385394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.385394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.385526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.386144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:53.386526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:53.387971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:53.406096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 235/475] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:53.409986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:53.430906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 241/476] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:53.484031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:53.535586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae426365-7bb8-4616-8f8e-8b4f46b22754 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:53.573366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d8381-a892-4417-be57-67e286417019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:53.574031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d8381-a892-4417-be57-67e286417019 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.574318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d8381-a892-4417-be57-67e286417019 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.574592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03d8381-a892-4417-be57-67e286417019 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.575238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03d8381-a892-4417-be57-67e286417019 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:53.576681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03d8381-a892-4417-be57-67e286417019 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:53.605668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 236/477] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:53 2022] POST /identity/v3/users => generated 249 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:53.756246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dce4dc-90a6-45d0-84e2-002582cf30e7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:53.783727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:53.872807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dce4dc-90a6-45d0-84e2-002582cf30e7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:53.908331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:53.909222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:53.909501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:53.909770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:53.910512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:53.911904 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:53.926148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.sql.core [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 01 04:04:53.926148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 01 04:04:53.926148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [parameters: {'id': '6c4faa682efe4802aa2a92a1013c6f0a', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 01 04:04:53.926148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81586) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result = self._query(query) Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application conn.query(q) Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result.read() Oct 01 04:04:53.942999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return ref.to_dict() Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 01 04:04:53.944642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application next(self.gen) Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application yield resource Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application next(self.gen) Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.session.rollback() Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.force_reraise() Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise self.value Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application session.commit() Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 01 04:04:53.945762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._prepare_impl() Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.session.flush() Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._flush(objects) Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application compat.raise_( Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application flush_context.execute() Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rec.execute(self) Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 01 04:04:53.947373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 01 04:04:53.949428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result = self._query(query) Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application conn.query(q) Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result.read() Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 01 04:04:53.954494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [parameters: {'id': '6c4faa682efe4802aa2a92a1013c6f0a', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:53.956313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 01 04:04:53.957667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:53.958645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 242/478] 10.222.0.26 () {60 vars in 1131 bytes} [Sat Oct 1 04:04:53 2022] POST /identity/v3/roles => generated 144 bytes in 190 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:53.958720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d8381-a892-4417-be57-67e286417019 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:54.072031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03d8381-a892-4417-be57-67e286417019 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:54.085787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:54.110167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a24ef-38e5-4556-85e2-2ba097344e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:54.110813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a24ef-38e5-4556-85e2-2ba097344e21 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:54.111093 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a24ef-38e5-4556-85e2-2ba097344e21 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:54.111365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a24ef-38e5-4556-85e2-2ba097344e21 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:54.122385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512a24ef-38e5-4556-85e2-2ba097344e21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-512a24ef-38e5-4556-85e2-2ba097344e21 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:54.127649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:54.129752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:54.132216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 237/479] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:04:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:54.140982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:54.229235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:54.253365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296f246f-bdf1-4aa6-bebc-49302bdc425b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:54.304341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2443d8e-6af6-49a9-8e8c-465da6c644c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:54.305040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2443d8e-6af6-49a9-8e8c-465da6c644c0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:54.305326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2443d8e-6af6-49a9-8e8c-465da6c644c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:54.305599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2443d8e-6af6-49a9-8e8c-465da6c644c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:54.306271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2443d8e-6af6-49a9-8e8c-465da6c644c0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:54.306572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2443d8e-6af6-49a9-8e8c-465da6c644c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:54.307941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2443d8e-6af6-49a9-8e8c-465da6c644c0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:54.326224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 243/480] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:55.682543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a767e88-b241-4313-91f0-c7115c5c5134 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:55.683057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a767e88-b241-4313-91f0-c7115c5c5134 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:55.683132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a767e88-b241-4313-91f0-c7115c5c5134 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:55.684678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 238/481] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:55.690425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121d74d-f388-4492-80ac-0410c4d2e051 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:55.690425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121d74d-f388-4492-80ac-0410c4d2e051 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:55.690425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121d74d-f388-4492-80ac-0410c4d2e051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:55.717870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f121d74d-f388-4492-80ac-0410c4d2e051 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:55.719058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f121d74d-f388-4492-80ac-0410c4d2e051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:55.808596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 244/482] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:55.938516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b478ee-247c-4707-b8f1-7417fe8cdd59 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:55.938857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b478ee-247c-4707-b8f1-7417fe8cdd59 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:55.939062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b478ee-247c-4707-b8f1-7417fe8cdd59 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:55.940500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 239/483] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:55.946113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab3342b-2754-4075-9b7f-feed219d0161 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:55.946642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab3342b-2754-4075-9b7f-feed219d0161 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:55.947110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab3342b-2754-4075-9b7f-feed219d0161 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:55.966038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-aab3342b-2754-4075-9b7f-feed219d0161 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:55.967503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-aab3342b-2754-4075-9b7f-feed219d0161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:56.050339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 245/484] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:56.296782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b478ee-247c-4707-b8f1-7417fe8cdd59 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:56.366987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab3342b-2754-4075-9b7f-feed219d0161 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:56.382719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:56.446685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b478ee-247c-4707-b8f1-7417fe8cdd59 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:56.490876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8753fe-5cd9-440f-bf42-49bab3b6b2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:56.492380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8753fe-5cd9-440f-bf42-49bab3b6b2ad None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.492662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8753fe-5cd9-440f-bf42-49bab3b6b2ad None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.493282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8753fe-5cd9-440f-bf42-49bab3b6b2ad None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.506435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab3342b-2754-4075-9b7f-feed219d0161 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:56.511108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8753fe-5cd9-440f-bf42-49bab3b6b2ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ed8753fe-5cd9-440f-bf42-49bab3b6b2ad None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:56.516750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:56.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:56.518690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 240/485] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:04:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:56.520399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed2f2c-0190-4787-8771-0346489976f5 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.520691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed2f2c-0190-4787-8771-0346489976f5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.520983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed2f2c-0190-4787-8771-0346489976f5 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.523031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 241/486] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:56.527286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4960bd2-ba3c-4a38-a89b-0587204f8b03 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.527286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4960bd2-ba3c-4a38-a89b-0587204f8b03 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.527286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4960bd2-ba3c-4a38-a89b-0587204f8b03 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.529058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 242/487] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:56.529828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4960bd2-ba3c-4a38-a89b-0587204f8b03 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:56.554485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d949baa-f418-423e-8904-dbf2cf16af72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:56.555190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d949baa-f418-423e-8904-dbf2cf16af72 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.555469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d949baa-f418-423e-8904-dbf2cf16af72 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.555737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d949baa-f418-423e-8904-dbf2cf16af72 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.563331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:56.566827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d949baa-f418-423e-8904-dbf2cf16af72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-5d949baa-f418-423e-8904-dbf2cf16af72 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:56.571624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:56.572555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:56.573429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 246/488] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:04:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:56.574579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0591b0-37ba-46bf-b822-52bbd1174d29 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.574579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0591b0-37ba-46bf-b822-52bbd1174d29 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.574866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0591b0-37ba-46bf-b822-52bbd1174d29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.593463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-6f0591b0-37ba-46bf-b822-52bbd1174d29 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:56.594722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6f0591b0-37ba-46bf-b822-52bbd1174d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:56.643605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4960bd2-ba3c-4a38-a89b-0587204f8b03 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:56.659738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:56.675493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 247/489] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:56.675493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a82f9-0e6d-4a4f-bb58-129a18d05f87 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.675493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a82f9-0e6d-4a4f-bb58-129a18d05f87 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.675493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a82f9-0e6d-4a4f-bb58-129a18d05f87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.691462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:56.692231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.692573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.692959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.693147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-294a82f9-0e6d-4a4f-bb58-129a18d05f87 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:56.693681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:56.693985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:56.695427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:56.695595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-294a82f9-0e6d-4a4f-bb58-129a18d05f87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:56.704458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 243/490] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:04:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:56.705413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:56.779038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 248/491] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:56.780340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a82f9-0e6d-4a4f-bb58-129a18d05f87 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:56.835408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:56.845034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9465f1e-1481-4aca-a8ba-b4dc390a3196 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:56.890466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a82f9-0e6d-4a4f-bb58-129a18d05f87 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:56.894704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:56.896489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.897093 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.901266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.902186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:56.902689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:56.904257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:56.922941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 244/492] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:04:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:56.926424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:56.941510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:56.955558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:56.957516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:56.958238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:56.960471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:56.982874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:56.985048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:57.014527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:57.015481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:57.016338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:57.016338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:57.016338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:57.018340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 249/493] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:57.024202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.099322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a358a76e-e489-4161-91b4-a184f9db4870 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.132911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.133566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.133847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.134123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.145053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.145835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bc6810-4bc2-495a-93b0-e4b977053e00 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.146495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:57.161214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:57.162120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:57.163010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:57.163010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:57.163010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:57.163010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 245/494] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:57.163603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.173961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.186425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03550df3-b78a-49a9-91cd-f905863097d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.187031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03550df3-b78a-49a9-91cd-f905863097d7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.187342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03550df3-b78a-49a9-91cd-f905863097d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.187615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03550df3-b78a-49a9-91cd-f905863097d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.188259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03550df3-b78a-49a9-91cd-f905863097d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:57.188564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03550df3-b78a-49a9-91cd-f905863097d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.189947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03550df3-b78a-49a9-91cd-f905863097d7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.202662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 250/495] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:04:57.204111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd41ce5-f06b-45ef-b977-b7b789506b2b None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.204396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd41ce5-f06b-45ef-b977-b7b789506b2b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.205073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd41ce5-f06b-45ef-b977-b7b789506b2b None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.206407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 251/496] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:04:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:04:57.209941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd41ce5-f06b-45ef-b977-b7b789506b2b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.250656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.263321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf150e5-11f2-4d0a-b7c1-a68f6d503aef None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.296861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.297515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.297796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.298070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.308226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:57.311295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd41ce5-f06b-45ef-b977-b7b789506b2b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:57.313388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:57.314338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:57.315123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 246/497] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:57.316346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.346542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.347160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.347436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.347708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.358074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:04:57.362678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:57.363726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 252/498] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:57.364724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.384906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.418785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e19bf3-56fc-4925-b879-b30fde8b6021 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.452546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e579cd1b-4d29-4fb9-89f6-06949af34f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.453255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e579cd1b-4d29-4fb9-89f6-06949af34f3b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.453542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e579cd1b-4d29-4fb9-89f6-06949af34f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.453819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e579cd1b-4d29-4fb9-89f6-06949af34f3b None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.454469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e579cd1b-4d29-4fb9-89f6-06949af34f3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:57.454775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e579cd1b-4d29-4fb9-89f6-06949af34f3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.456144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e579cd1b-4d29-4fb9-89f6-06949af34f3b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.459041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.463412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04270013-20d9-480b-b90f-c59fb2053662 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.467785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 247/499] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:57.469061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a6bf5d-8286-4fda-97fa-a021f8b57662 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.469351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a6bf5d-8286-4fda-97fa-a021f8b57662 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.469624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a6bf5d-8286-4fda-97fa-a021f8b57662 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.486915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-06a6bf5d-8286-4fda-97fa-a021f8b57662 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:04:57.488135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-06a6bf5d-8286-4fda-97fa-a021f8b57662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:04:57.499292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.499920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.500199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.500478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.508335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.509812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:04:57.522721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:57.523772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 253/500] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:57.525025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.561010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 248/501] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:04:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:04:57.561853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a6bf5d-8286-4fda-97fa-a021f8b57662 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.629663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6c24d9-cc72-452a-acd1-1e7cee8fc324 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.631569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.665819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.666429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.666722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.666995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.667683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:57.667983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.669389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.674482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a6bf5d-8286-4fda-97fa-a021f8b57662 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.678883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 254/502] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:57.679973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.705196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.713400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.714074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.714357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.714553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.715208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:57.715522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.716896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.728977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 249/503] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:57.728977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.797526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1867be19-20af-448a-8984-4f0bd5fb763b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.832612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.833252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a865ff0-e252-4b53-b9ca-562fc4e148e8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.833357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.833532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.833804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] PATH_INFO: `/v3/users/glance` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.841325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.842738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.853747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:04:57.853946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:57.855008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 255/504] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/users/glance => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:57.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.871813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.872457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.872690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.872988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.873707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:57.874009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:57.875456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:57.916484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 250/505] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:57.916484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:57.935652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:57.954853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f88f72e-3612-4961-9104-59fc2231da1a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:57.987445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:57.988061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:57.988336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:57.988603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:57.999951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.001884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:58.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:58.018767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:58.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:58.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:58.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:58.019780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a255a27-4a09-4964-8a29-f0675fbae4d9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.019859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 256/506] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:58.019859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.053489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.054120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.054401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.054675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.065121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.066600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:58.081249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:58.082325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:58.083207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:58.083207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:58.083207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:58.083207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 251/507] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:58.083207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.115780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.123575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce992fe-4958-4c33-9030-20e24e3920cd None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.152527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.157542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.158165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.158700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.158983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.170013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.171460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.182947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e611bfa6-f83b-4b54-82e7-3b986d063d7b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:58.186144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:58.187041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:58.187874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:58.187874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:58.187874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:58.187874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 257/508] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:58.188471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.218758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.219364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.219665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.219945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.220665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:58.221006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.222435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.239585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 252/509] 10.222.0.26 () {56 vars in 1137 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:58.241493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.289931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800163e-1d43-4370-a1cd-80aa0de01dd3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.328254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.328850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.329162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.329426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.332822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.345601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be81525-43f9-4790-8a55-7345a868e28e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.379836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.380441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.380720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.381619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.382273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:58.382437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.382592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.383953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.395190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 253/510] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:58.395957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.398734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.400200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.493410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a250b01-4058-4262-8417-bd1b2a88c8a1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.498268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 258/511] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:58.499100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.532980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.533858 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.534138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.534413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.535054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:58.535361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.536726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.548331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 254/512] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:58.549176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.560360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.585559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.602352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89f5f99-e28d-4513-9216-9a45fa72097b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.640255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.641030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.641316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.641589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.642231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:58.642533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.643887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.653217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec32413-fc3c-42a1-b5df-2654a705015b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.658291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 259/513] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:58.659349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.687695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.688295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.688573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.688842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.763429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.764947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.770905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e437c-df3c-4f56-8a57-4e6943320518 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:58.776102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.822292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:58.823293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:58.823750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:58.824203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:58.836903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:58.839185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:04:58.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:58.857908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:58.858841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 260/514] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/users/cinder => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:58.859429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.863698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:58.899521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 255/515] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:58.900589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:58.966229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cfad07-f8e4-4e83-939a-aa11b0579c70 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.005702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.006397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.006678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.006952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.008203 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e7e9e-3443-474d-bd04-d4411e59fc01 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.019165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.019606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.021429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:59.036491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:59.038276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:59.039545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:59.039545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:59.039545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:59.039545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 261/516] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:59.039545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.051046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.051674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.051948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.052213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.059626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.061071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:04:59.075564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:59.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:59.077949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 256/517] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:04:58 2022] GET /identity/v3/users/neutron => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:59.081934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.096354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.139690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3e4d7b-8fac-488d-8ae1-a89dec7371d9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.174617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.175220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.175502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.175778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.176426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.177944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.180090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5d481c-dc0b-4e71-8eb8-cc3c82b77592 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.199227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 262/518] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:04:59 2022] POST /identity/v3/users => generated 247 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:04:59.200035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.214790 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.215419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.215701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.215974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.216739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:59.217084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.218586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.238438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 257/519] 10.222.0.26 () {56 vars in 1137 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:59.239311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.270962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.328075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8516db3-9139-4c6f-a0da-6b1ce83aa111 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.342900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.353909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7abfa0-7f4d-4c8d-9752-6ccf4cc5bec3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.370540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.371324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.371604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.371869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.385764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.387527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.392644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.393333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.394046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.394329 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.394974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:59.395664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.397213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:04:59.402787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:59.403709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:04:59.404561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:04:59.404561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:04:59.404561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:04:59.404561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 263/520] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:59.404782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.414070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 258/521] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:59.414070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.499501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.518737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850439b2-34e6-49be-8bda-ec720c5ac85f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.526366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176ceb9f-bad0-4bcd-b224-5130edfd4f3b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.555175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.555797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.556080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.556355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.563155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.563965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.564383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.564802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.565826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:59.566272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.568088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.608065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 264/522] 10.222.0.26 () {56 vars in 1139 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:59.609163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.633206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.635117 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.637342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.720547 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.726477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e78e850-474a-4148-b0bb-4646e6166da9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.743234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 259/523] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:04:59.743234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.767946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.768544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.768821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.769322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.769749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:04:59.770054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.771408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.788315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 265/524] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:04:59.788315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.847186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161784-cea0-4d3b-9d8d-6518173147ac None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.877759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:04:59.883440 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.884143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.884846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.884846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.891915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:04:59.893479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:04:59.897323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303d6e6-bd88-4c22-bbe7-d849215a0bf4 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:04:59.906779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:04:59.907824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 260/525] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:04:59.914591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:04:59.942332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:04:59.943025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:04:59.943914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:04:59.944248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:04:59.961544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.018223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.019258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc995b8-ab10-4025-a281-2510a4028bbf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.019675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.056994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.057868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.058148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.058421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.070396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.071849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:00.086392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:00.091931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:00.093143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:00.093143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:00.093143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:00.093143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 261/526] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:00.093143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.112333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.139838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 266/527] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:04:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:05:00.139838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.195662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.200912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b189ac5b-73bc-440f-bf19-efa24ec580bc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.236649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.237299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.237577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.237847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.243629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad765a7-e8df-4252-ad2e-93d1218ca957 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.245072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.246514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:00.258161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:00.259185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 262/528] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:00.260038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.286865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.287816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.288243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.288662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.289525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:00.290046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.291715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.312191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 267/529] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:00.313200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.361814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.363073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f932a8-77b1-4f4c-9c4f-7ac318b621d3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.395980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.397196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.397732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.398045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.398693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:00.399002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.400500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.415846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 263/530] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:00.416705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.434548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c99b09-d80f-4bf7-bd9d-92cb57337175 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.439004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.470991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.471819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.472155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.472575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.483934 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.485669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:00.500968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:00.501993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:00.502829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:00.502829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:00.502829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:00.507656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 268/531] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:00.508562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.534093 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6798bf-56bc-4f7b-82be-1f0c70513e5c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.570728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.571518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.571804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.572081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.573219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:00.573679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.575112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.593216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 264/532] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:00.598271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.612873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.621042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f2bdb-6049-466c-b999-71993e309a29 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.660099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.660715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.661320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.661593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.662993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=901b9bb47c2f40c2b1b4cfe7886a1167) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:00.663301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.664902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.680964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 269/533] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/role_assignments?user.id=901b9bb47c2f40c2b1b4cfe7886a1167&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:00.680964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.700469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.737152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356588b7-2447-4e9d-9326-cb41672234e2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.786699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.787818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.788171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.788513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.800261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.801982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.813253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d592515-fc11-48b7-a676-38c7327e72e5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:00.817377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:00.818257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:00.819634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 265/534] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:00.824561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.858328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:00.858943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:00.859220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:00.859505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:00.860435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:00.860739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:00.862153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:00.876037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 270/535] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:00.876897 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:00.904129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:00.960359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd3711d-e244-4ad5-910e-8636171fdb7d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:01.000617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e08ee-2bfd-40d3-aadf-8669d89f8bee None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:01.004687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:01.005725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.006269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.006737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.007674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=170fe0394f3d4472a1380481cfe2cc28) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:01.008244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:01.010328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:01.017747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 266/536] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/role_assignments?user.id=170fe0394f3d4472a1380481cfe2cc28&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:01.019084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:01.023149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:01.045218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:01.046100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.046381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.046651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.047349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:01.047654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:01.049574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:01.067341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 271/537] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:01.073216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:01.216209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae2cd46-a34f-49d4-8e31-314c67522fdd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:01.253465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:01.269230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:01.270001 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.270259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.270534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.278504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:01.279777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:01.281728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f767208c-753c-4d2d-9100-5fff3d257d01 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:01.292785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:01.293685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:01.294509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:01.298685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 267/538] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:01.300323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:01.322096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c306b-89eb-4587-bd3a-9e18194260a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:01.323104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c306b-89eb-4587-bd3a-9e18194260a2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.323608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c306b-89eb-4587-bd3a-9e18194260a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.324138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c306b-89eb-4587-bd3a-9e18194260a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.332752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c306b-89eb-4587-bd3a-9e18194260a2 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=52b5d64c999a4f16afec678df3025529) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:01.333640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c306b-89eb-4587-bd3a-9e18194260a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:01.335585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c306b-89eb-4587-bd3a-9e18194260a2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:01.342452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 272/539] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:01 2022] GET /identity/v3/role_assignments?user.id=52b5d64c999a4f16afec678df3025529&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 259 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:01.396034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:01.427731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb44d62-4699-45bf-959c-e8aaacdfd456 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:01.466338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a251b05-cb16-42db-92e6-a1db17011544 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.467035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a251b05-cb16-42db-92e6-a1db17011544 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.467565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a251b05-cb16-42db-92e6-a1db17011544 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.469430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 273/540] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:01.475786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934814ce-bfd1-4759-8877-77a97eb46ada None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.476161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934814ce-bfd1-4759-8877-77a97eb46ada None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.476508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934814ce-bfd1-4759-8877-77a97eb46ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.478835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:01.479550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.479981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.480398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.481302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:01.481684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:01.483370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:01.497555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 268/541] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:01.509434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:01.515700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-934814ce-bfd1-4759-8877-77a97eb46ada None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:01.516970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-934814ce-bfd1-4759-8877-77a97eb46ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:01.648733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 274/542] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:05:01.675437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa810089-0a20-4143-813f-8962bd767393 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:01.702979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:01.720191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:01.721040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:01.721457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:01.721889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:01.722750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=16aab142c0a7445483caf19178d56062) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:01.723195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:01.724809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:01.734498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 269/543] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:01 2022] GET /identity/v3/role_assignments?user.id=16aab142c0a7445483caf19178d56062&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 259 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:02.010455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934814ce-bfd1-4759-8877-77a97eb46ada None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:02.090761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:02.134138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934814ce-bfd1-4759-8877-77a97eb46ada None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:02.184239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:02.184883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:02.185192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:02.185467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:02.197323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:02.202138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:02.203002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:02.203787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 275/544] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:02.209396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:02.322165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768ab2c8-d8d2-4ee1-be3f-9eef656987b8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:02.360420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:02.361415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:02.361913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:02.362421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:02.363263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:02.363765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:02.365401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:02.379517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:02.379904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 270/545] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:02.532586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3fcc74-2e32-47c1-b698-8ae4075b0cdf None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:02.571417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:02.582191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:02.582761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:02.583037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:02.583308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:02.598814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:02.601277 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:02.621513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:02.622409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:02.623238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:02.623238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:02.623238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:02.623238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 276/546] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:02.628495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:02.705633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:02.750766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5527a3f9-179c-4a87-8aa3-533480c24bf8 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:02.785513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:02.786519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:02.786802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:02.787608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:02.788482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:02.788787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:02.790207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:02.803620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 271/547] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:02.809294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:02.951805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b8a204-315f-4ea2-b5b4-9214a8fac5d9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:03.001621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:03.002254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:03.002536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:03.002810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] PATH_INFO: `/v3/users/swift` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:03.011109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:03.012659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:03.028433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:03.029910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:03.031032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 277/548] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:02 2022] GET /identity/v3/users/swift => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:03.038005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:03.155650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9affd15-631d-44f6-ba0a-58cc29acaf7a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:03.177317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:03.192696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d264686-7119-4ef3-a633-d56b8e1fd1ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:03.193392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d264686-7119-4ef3-a633-d56b8e1fd1ce None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:03.193672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d264686-7119-4ef3-a633-d56b8e1fd1ce None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:03.193946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d264686-7119-4ef3-a633-d56b8e1fd1ce None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:03.194666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d264686-7119-4ef3-a633-d56b8e1fd1ce None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:03.194968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d264686-7119-4ef3-a633-d56b8e1fd1ce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:03.196411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d264686-7119-4ef3-a633-d56b8e1fd1ce None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:03.215291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 272/549] 10.222.0.26 () {56 vars in 1135 bytes} [Sat Oct 1 04:05:03 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:03.222211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:03.294382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b964a4-9283-4db4-9b65-c37da6558515 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:03.295037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b964a4-9283-4db4-9b65-c37da6558515 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:03.295512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b964a4-9283-4db4-9b65-c37da6558515 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:03.297728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 273/550] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:03.303812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b31b80-e6b5-48c4-8864-8e23e61ef76c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:03.304354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b31b80-e6b5-48c4-8864-8e23e61ef76c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:03.304822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b31b80-e6b5-48c4-8864-8e23e61ef76c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:03.306875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:03.328810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-29b31b80-e6b5-48c4-8864-8e23e61ef76c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:03.330249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-29b31b80-e6b5-48c4-8864-8e23e61ef76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:03.414217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 274/551] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:05:03.438336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48b8efc-3d8e-424a-ac5b-ff7a7f6c50df None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:03.488596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:03.489199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:03.489484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:03.489759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:03.579496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:03.581227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:03.638973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:03.711405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 278/552] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:05:03.716954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b31b80-e6b5-48c4-8864-8e23e61ef76c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:03.766694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:03.900320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:03.902818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b31b80-e6b5-48c4-8864-8e23e61ef76c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:03.960739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c020c0d-49f1-48be-a2d3-f88a371ef6a3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:03.964281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cbf905-418f-4f9f-963b-4ed052bc5ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:03.965200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cbf905-418f-4f9f-963b-4ed052bc5ec8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:03.965773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cbf905-418f-4f9f-963b-4ed052bc5ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:03.966244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cbf905-418f-4f9f-963b-4ed052bc5ec8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:03.979334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cbf905-418f-4f9f-963b-4ed052bc5ec8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:03.981645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cbf905-418f-4f9f-963b-4ed052bc5ec8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-85cbf905-418f-4f9f-963b-4ed052bc5ec8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:04.001796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:04.003587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:04.006115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:04.006115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:04.006115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:04.006908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1d5566-ebe8-4a75-8ac5-8c56e520410d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.007805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 275/553] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:04.007921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1d5566-ebe8-4a75-8ac5-8c56e520410d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.008394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1d5566-ebe8-4a75-8ac5-8c56e520410d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.008862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1d5566-ebe8-4a75-8ac5-8c56e520410d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.009108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4177c53a-84a4-4a97-85f4-8397517a610c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.009396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4177c53a-84a4-4a97-85f4-8397517a610c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.009675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4177c53a-84a4-4a97-85f4-8397517a610c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.015436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 276/554] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:04 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:04.016237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4177c53a-84a4-4a97-85f4-8397517a610c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.022881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1d5566-ebe8-4a75-8ac5-8c56e520410d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-5b1d5566-ebe8-4a75-8ac5-8c56e520410d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:04.028573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:04.029639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:04.030451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:04.033839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 279/555] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:04.039563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5cd1b3-048b-4e76-9e0a-05d9f106244e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.041080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5cd1b3-048b-4e76-9e0a-05d9f106244e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.042386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5cd1b3-048b-4e76-9e0a-05d9f106244e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.060236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:04.073753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8c5cd1b3-048b-4e76-9e0a-05d9f106244e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:04.075017 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8c5cd1b3-048b-4e76-9e0a-05d9f106244e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:04.154234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4177c53a-84a4-4a97-85f4-8397517a610c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.157028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 280/556] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:04.158117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5cd1b3-048b-4e76-9e0a-05d9f106244e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.198645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5773c6-e530-4e7a-ac1a-feaa5add9f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.200758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5773c6-e530-4e7a-ac1a-feaa5add9f85 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.201287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5773c6-e530-4e7a-ac1a-feaa5add9f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.201762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5773c6-e530-4e7a-ac1a-feaa5add9f85 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.202608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5773c6-e530-4e7a-ac1a-feaa5add9f85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:04.203130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5773c6-e530-4e7a-ac1a-feaa5add9f85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:04.205233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5773c6-e530-4e7a-ac1a-feaa5add9f85 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.221452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 277/557] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:04.223373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46c96-2561-4232-8e70-3830c7de0e6e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.223863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46c96-2561-4232-8e70-3830c7de0e6e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.224332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f46c96-2561-4232-8e70-3830c7de0e6e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.226383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 278/558] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:04.227436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f46c96-2561-4232-8e70-3830c7de0e6e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.239885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:04.296644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5cd1b3-048b-4e76-9e0a-05d9f106244e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.338745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6b7422-d386-4fee-aec3-d6f4bcfa0f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.339353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6b7422-d386-4fee-aec3-d6f4bcfa0f16 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.339630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6b7422-d386-4fee-aec3-d6f4bcfa0f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.339897 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6b7422-d386-4fee-aec3-d6f4bcfa0f16 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.340561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6b7422-d386-4fee-aec3-d6f4bcfa0f16 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:04.340862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6b7422-d386-4fee-aec3-d6f4bcfa0f16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:04.342232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6b7422-d386-4fee-aec3-d6f4bcfa0f16 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.343758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:04.357397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 281/559] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:04.357397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9ed01d-98e3-4813-a459-dc584239f54b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.357397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9ed01d-98e3-4813-a459-dc584239f54b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.357397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9ed01d-98e3-4813-a459-dc584239f54b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.367163 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f46c96-2561-4232-8e70-3830c7de0e6e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.375445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5c9ed01d-98e3-4813-a459-dc584239f54b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:04.377099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5c9ed01d-98e3-4813-a459-dc584239f54b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:04.411667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.413237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.414014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.414502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.424914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:04.426832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:04.445292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:04.446275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:04.447573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 279/560] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:04.448267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.467794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 282/561] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:04.469098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9ed01d-98e3-4813-a459-dc584239f54b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.536796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:04.573484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd10ca4-24f5-4df1-bf6c-c86a14672c8e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.592206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9ed01d-98e3-4813-a459-dc584239f54b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.610600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.611231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.611511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.611783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.619823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:04.623076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:04.624511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.630023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bba37ec-bc38-462e-81c8-b17a3d33bfee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.631208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bba37ec-bc38-462e-81c8-b17a3d33bfee None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.631659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bba37ec-bc38-462e-81c8-b17a3d33bfee None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.632004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bba37ec-bc38-462e-81c8-b17a3d33bfee None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:04.640226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:04.641436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:04.642837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:04.642837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:04.642837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:04.642837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 280/562] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:04.642837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.647175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bba37ec-bc38-462e-81c8-b17a3d33bfee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-4bba37ec-bc38-462e-81c8-b17a3d33bfee None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:04.652045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:04.653135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:04.653987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 283/563] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:04.654664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79d160d-f6fc-429e-b45c-40d7958f6d14 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.654868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79d160d-f6fc-429e-b45c-40d7958f6d14 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.655135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79d160d-f6fc-429e-b45c-40d7958f6d14 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.661174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 284/564] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:04 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:04.662266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79d160d-f6fc-429e-b45c-40d7958f6d14 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.755204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d09f46d-bb04-4301-a8af-8fd11ef0d59e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.767989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79d160d-f6fc-429e-b45c-40d7958f6d14 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.777061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:04.791748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.792362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.792642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.792943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.793639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:04.793948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:04.795484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.806112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556455f8-eabd-4f63-885f-9017b8d71e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.806712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556455f8-eabd-4f63-885f-9017b8d71e59 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.806991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556455f8-eabd-4f63-885f-9017b8d71e59 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.807261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556455f8-eabd-4f63-885f-9017b8d71e59 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.807564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 281/565] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:04.807907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556455f8-eabd-4f63-885f-9017b8d71e59 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:04.808209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556455f8-eabd-4f63-885f-9017b8d71e59 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:04.808346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.809572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556455f8-eabd-4f63-885f-9017b8d71e59 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.821363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 285/566] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:04.822409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d983a3-de01-49b5-974d-624d5af49887 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.822694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d983a3-de01-49b5-974d-624d5af49887 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.822964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d983a3-de01-49b5-974d-624d5af49887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.840059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c2d983a3-de01-49b5-974d-624d5af49887 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:04.841194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c2d983a3-de01-49b5-974d-624d5af49887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:04.882752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:04.907946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 286/567] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:04.909041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d983a3-de01-49b5-974d-624d5af49887 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:04.910414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f896a911-610c-4043-9eba-daadddeb1c79 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:04.947629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:04.948234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:04.948513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:04.948783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:04.949468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:04.949769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:04.951124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:04.963161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 282/568] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:04.963161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.003410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.018901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d983a3-de01-49b5-974d-624d5af49887 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.054343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.054957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.055240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.055513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.064516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9c1d43-7d7f-4c80-ae87-50ed32071edd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.065753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:05.071193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:05.072107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:05.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 287/569] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:05.074221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.105599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.106243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.106526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.106975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.107653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=3d69adfda7334567b03627a1d839f4f2) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:05.107965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.109438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.111634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.120975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 283/570] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:04 2022] GET /identity/v3/role_assignments?user.id=3d69adfda7334567b03627a1d839f4f2&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:05.120975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.205261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25c7389-7cfa-4d56-9ffe-ca095e2e9425 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.246756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.247468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.247681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.247954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] PATH_INFO: `/v3/users/nova` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.252220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccce51b-2409-4ef0-82b6-453118e632af None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.256378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.257890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.261034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:05.270547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:05.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:05.272835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:05.276617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 288/571] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:05.277469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.297630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.298374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.298582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.298801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.311702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.313272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:05.328593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:05.329697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:05.330590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:05.330590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:05.330590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:05.334835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 284/572] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:05.335214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.381467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.388484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f776ba-414a-4bd0-9b25-1ef23827d9af None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.424505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.425143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.425426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.425706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.426354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:05.426777 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.428759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.436732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed8a420-12fb-4ef2-a17e-b7e5f91121c1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.442084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 289/573] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:05.442964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.471166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.471795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.472076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.472348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.482882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:05.488051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:05.488988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:05.489835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 285/574] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:05.491109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.514985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.564390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd456d1-1f9a-4242-bfc5-11da24e043b6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.602416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356a858a-5b67-441e-aaf5-acf2289f996c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.606361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.607198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.607634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.608044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.608906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:05.609375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.610919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.630536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 290/575] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:05.631755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.641560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.642173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.642452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.642722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.643362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:05.643663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.645110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.651974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.658676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 286/576] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:05.659481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.737125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.749881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364c3a5a-e659-40ad-aab8-1e4e94ec6def None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.759097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d3165e-8405-4cac-85db-c092508eafd4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.786274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.787519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.787916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.788208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.795189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.795774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.796051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.796321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.797012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:05.797316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.798618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.798738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.800042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.807557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 287/577] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:05.812239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:05.820699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:05.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:05.822750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:05.822750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:05.822750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:05.822750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 291/578] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:05.822750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.896650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.918996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48df5f1-d895-4d60-8a50-1cf83f541b7a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.934644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124aa9d-caea-4226-a6b5-12a446abf21c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:05.954186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.954816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.955501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.955779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.964419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:05.966425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.967887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.970774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:05.971407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:05.971689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:05.971964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:05.980320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:05.981784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:05.983407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:05.984530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:05.985741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:05.985741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:05.985741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:05.989097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 288/579] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:05.989911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:05.997517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:05.998668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 01 04:05:05.999867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:06.008089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 292/580] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/users/glance => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:06.008089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.102294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd9f170-d338-42e6-abc9-0c7f42e24762 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.111406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e16b59-e6ea-44fc-9d41-7b0024216301 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.138098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.139919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.140188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.140768 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.141022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.150545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.151218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.151545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.151826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.152480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:06.152785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.154190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.155951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.158034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.171087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 293/581] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:06.171087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:06.178582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:06.179508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:06.180399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:06.180399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:06.180399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:06.180399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 289/582] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:06.181517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.227650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.280273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05b857-60e3-4a52-ba82-ce9923a7a86d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.283857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf8abed-d947-4594-ae93-dc6d281f9d0b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.317266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.317871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.318251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.318575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.318991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.319219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:06.319525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.319617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.319855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.320122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.320841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:06.320949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.321179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.322581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.333468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 294/583] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:06.333468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.342830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 290/584] 10.222.0.26 () {56 vars in 1137 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:06.346557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.388021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.450830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5fb3e-bd6b-438c-8ee6-575a20829a22 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.453969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405782e7-d1f3-4112-bac7-fafd08c498c0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.455617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.485859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.486490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.486774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.487106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.489466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.490610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.490889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.491168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.491800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:06.492103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.493517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.496220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.497282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:06.510643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:06.511612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:05:06.512549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:06.513757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 291/585] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:06.513757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.518525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 295/586] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/users/cinder => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:06.518525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.617566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d000070f-2e8d-4c63-9025-64d186826948 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.624879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c11120f-74fd-4fbd-ae8a-be67e5557631 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.635615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.653236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.653866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.654152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.654429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.661105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.663070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.664210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.665186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.665697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.666149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:06.674694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:06.675579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 292/587] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:06.676499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.683316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.685061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:06.702153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:06.703239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:06.704432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:06.704432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:06.704432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:06.707628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 296/588] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:06.708282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.709889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.793708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe68c677-f649-4530-8157-c08de2247cc1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.815798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e95d0-243e-4bbb-903b-9364132bab78 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.828749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.829415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.829702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.829990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.837973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.839423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:06.851187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:06.852168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 01 04:05:06.853552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:06.854095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:06.854095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:06.854095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:06.854305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 293/589] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/users/neutron => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:06.854305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.857254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:06.858031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:06.858348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:06.859794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:06.877776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 297/590] 10.222.0.26 () {56 vars in 1137 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:06.878606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:06.902501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.947209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:06.969704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5f1542-3dd0-4f50-bb07-fede665fbff9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:06.991448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b98fff-4845-43ee-9383-0cab3e4db3b4 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.004657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.005300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.005601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.005882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.006568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:07.006872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.008303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.020393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 294/591] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:07.021862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.032977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.034004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.034383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.034748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.035500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:07.035885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.038214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.054515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 298/592] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:07.054515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.128826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d60001b-ce2f-4d03-bbda-d16e99a36dc5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.138190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:07.165103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcfc897-7162-45d6-896a-7f3b113276f0 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.166291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.167062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.167352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.167628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.169210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:07.169526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.170968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.188364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:07.198649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 295/593] 10.222.0.26 () {56 vars in 1139 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:07.199602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.213357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.214000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.214279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.214592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.227027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.228406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:07.243894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:07.244857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:07.250479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:07.250479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:07.250479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:07.250479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 299/594] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:07.250479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.309887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df53c4d-8972-4cd2-96e6-dada549b9491 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.343124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.343774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.344050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.344321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/901b9bb47c2f40c2b1b4cfe7886a1167/roles/8f8d9a3eff6c47faa94d854a3f42be35` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.353992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec927061-5cd5-4d3a-a177-ecdb7e6b936d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.359249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=901b9bb47c2f40c2b1b4cfe7886a1167, role_id=8f8d9a3eff6c47faa94d854a3f42be35)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.360678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.373894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 296/595] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:07 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/901b9bb47c2f40c2b1b4cfe7886a1167/roles/8f8d9a3eff6c47faa94d854a3f42be35 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:05:07.374679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.382597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:07.388688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.389711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.390124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.390599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.404671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.406548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:07.421337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:07.422317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:07.423237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 300/596] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:07.424036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.459574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:07.531027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb9204b-706a-40f9-98dd-0a2cb03b5565 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.545490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e164c440-0d69-4da7-9973-64044f75a0f7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.572080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.572875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.573268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.573633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.584872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.586392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0127cfb1-c94e-4529-b805-7f81986c5dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.586520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.587158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0127cfb1-c94e-4529-b805-7f81986c5dff None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.587489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0127cfb1-c94e-4529-b805-7f81986c5dff None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.587837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0127cfb1-c94e-4529-b805-7f81986c5dff None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.588556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0127cfb1-c94e-4529-b805-7f81986c5dff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:07.588956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0127cfb1-c94e-4529-b805-7f81986c5dff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.590393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0127cfb1-c94e-4529-b805-7f81986c5dff None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:07.601765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:07.602930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:07.603914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:07.603914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:07.603914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:07.604082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 301/597] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:07.604436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0488753-79a2-4a37-8724-b1e6b135b5ef None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.604797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0488753-79a2-4a37-8724-b1e6b135b5ef None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.605136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 297/598] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:07.605229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0488753-79a2-4a37-8724-b1e6b135b5ef None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.606150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.606618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 302/599] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:07.617167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0488753-79a2-4a37-8724-b1e6b135b5ef None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.672128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:07.714926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce90d7be-51b4-4761-8f02-ab6b0f7358f3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.716895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:07.730528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0488753-79a2-4a37-8724-b1e6b135b5ef None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.757239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d9317b-fdf5-46cb-8039-81d2cab03dcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.757849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d9317b-fdf5-46cb-8039-81d2cab03dcb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.758127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d9317b-fdf5-46cb-8039-81d2cab03dcb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.758396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d9317b-fdf5-46cb-8039-81d2cab03dcb None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.759081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d9317b-fdf5-46cb-8039-81d2cab03dcb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:07.759384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d9317b-fdf5-46cb-8039-81d2cab03dcb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.760785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d9317b-fdf5-46cb-8039-81d2cab03dcb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.769408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.770636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.771838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.772594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.773487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 298/600] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:07.776673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277efc0-be33-4041-b93f-37734b0e65f1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.776969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277efc0-be33-4041-b93f-37734b0e65f1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.777243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277efc0-be33-4041-b93f-37734b0e65f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.788299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.790062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.796366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b277efc0-be33-4041-b93f-37734b0e65f1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:07.797595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b277efc0-be33-4041-b93f-37734b0e65f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:07.804109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:07.805210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:07.806301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:07.812707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 303/601] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:07.813809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.882321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 299/602] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:07.883145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277efc0-be33-4041-b93f-37734b0e65f1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:07.934697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa319bc2-d018-4b9a-96ab-aa225cd07feb None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:07.949211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:07.972116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be201957-6334-4166-99f2-08097b2e156d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:07.973006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be201957-6334-4166-99f2-08097b2e156d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:07.973431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be201957-6334-4166-99f2-08097b2e156d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:07.974077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be201957-6334-4166-99f2-08097b2e156d None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:07.976347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be201957-6334-4166-99f2-08097b2e156d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:07.976814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be201957-6334-4166-99f2-08097b2e156d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:07.978369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be201957-6334-4166-99f2-08097b2e156d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:07.993305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 304/603] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:07.994363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be201957-6334-4166-99f2-08097b2e156d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.003341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:08.014613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277efc0-be33-4041-b93f-37734b0e65f1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.058102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.058692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.058975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.059249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/170fe0394f3d4472a1380481cfe2cc28/roles/8f8d9a3eff6c47faa94d854a3f42be35` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.080024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=170fe0394f3d4472a1380481cfe2cc28, role_id=8f8d9a3eff6c47faa94d854a3f42be35)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:08.082474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:08.099974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 300/604] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:07 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/170fe0394f3d4472a1380481cfe2cc28/roles/8f8d9a3eff6c47faa94d854a3f42be35 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:05:08.101118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.156012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be201957-6334-4166-99f2-08097b2e156d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.223411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.224224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.225444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.225826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.227222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:08.227896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:08.229816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:08.247569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 305/605] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:08.249325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.279424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e192d0-9b70-4bc0-adf5-0e3bdc2d38b4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.295061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:08.326846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.328176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.328461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.328739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.337748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:08.339350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:08.352176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:08.353309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:08.354396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:08.355621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 301/606] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:08.356624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.363685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:08.393206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e9249e-7375-4293-a9bd-5eaf3a0b9f6b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.434365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.435987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.436432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.436848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.449876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:08.457660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:08.458950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:08.460000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:08.462349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 306/607] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:08.463856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.494200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bbbaf3-76e9-4898-8e9b-70f37fbf42e7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.543263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.544307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.544846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.545351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/52b5d64c999a4f16afec678df3025529/roles/8f8d9a3eff6c47faa94d854a3f42be35` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.561644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=52b5d64c999a4f16afec678df3025529, role_id=8f8d9a3eff6c47faa94d854a3f42be35)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:08.563536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:08.580194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 302/608] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:08 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/52b5d64c999a4f16afec678df3025529/roles/8f8d9a3eff6c47faa94d854a3f42be35 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:05:08.580906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:08.582675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.610959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc6596e-a2fd-40fa-88b2-433ea636950b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.686188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:08.703162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.704154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.704581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.705046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.705912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:08.706388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:08.708120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:08.736160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 307/609] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:08.742080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.775500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41272f0-54f0-4034-84d5-1e3b656a3aff None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.818699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.819575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.820064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.820485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.821360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:08.821918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:08.823448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:08.833086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 303/610] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:08.839801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:08.877754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c54575-f64f-4d95-915e-2cca12c1f2ed None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.914541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:08.915202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:08.915484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:08.915760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/16aab142c0a7445483caf19178d56062/roles/8f8d9a3eff6c47faa94d854a3f42be35` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:08.930719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=16aab142c0a7445483caf19178d56062, role_id=8f8d9a3eff6c47faa94d854a3f42be35)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:08.932207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:08.946856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:08.953449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 308/611] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:08 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/16aab142c0a7445483caf19178d56062/roles/8f8d9a3eff6c47faa94d854a3f42be35 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:05:08.955396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841109bf-40b8-4387-ac92-6ab5269d91c5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:08.971595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:09.008297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:09.009398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:09.009882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:09.010340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:09.027926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:09.029799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:09.051670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:09.052819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:09.053920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:09.053920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:09.053920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:09.055448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 304/612] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:09.061984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:09.187189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5c03a-6e03-40d5-8e64-2e415d8f0fd8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:09.226152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:09.226766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:09.227048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:09.227324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:09.227974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:09.228278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:09.229671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:09.242883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 309/613] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:09.247465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:09.391334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8632c-8ba1-47cf-bab7-dc7f97d2d921 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:09.440480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:09.441069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:09.441362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:09.441639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:09.449571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:09.451023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:09.462815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:09.463813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:09.464797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:09.468561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 305/614] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:09 2022] GET /identity/v3/users/swift => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:09.469952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:09.532215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:09.575787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3854e478-0318-4df5-8cac-78fd2d89756c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:09.612679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b7020-06e3-481b-a09a-f06c9e14a8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:09.613344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b7020-06e3-481b-a09a-f06c9e14a8e5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:09.613628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b7020-06e3-481b-a09a-f06c9e14a8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:09.613906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b7020-06e3-481b-a09a-f06c9e14a8e5 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:09.614659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b7020-06e3-481b-a09a-f06c9e14a8e5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:09.615521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b7020-06e3-481b-a09a-f06c9e14a8e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:09.616984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b7020-06e3-481b-a09a-f06c9e14a8e5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:09.641043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 310/615] 10.222.0.26 () {56 vars in 1135 bytes} [Sat Oct 1 04:05:09 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:09.642668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:09.678944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:09.745110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4aa334-d25b-4f19-be48-efd9352c274f None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:09.745315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4aa334-d25b-4f19-be48-efd9352c274f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:09.745576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4aa334-d25b-4f19-be48-efd9352c274f None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:09.749038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 311/616] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:09.754873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e38052a-b65d-4760-9f0d-9dea8e13f189 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:09.755160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e38052a-b65d-4760-9f0d-9dea8e13f189 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:09.755437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e38052a-b65d-4760-9f0d-9dea8e13f189 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:09.770588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947e7bb1-32da-4a88-9cf3-3a7c3bd3d94b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:09.790902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5e38052a-b65d-4760-9f0d-9dea8e13f189 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:09.792129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5e38052a-b65d-4760-9f0d-9dea8e13f189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:09.819586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:09.820521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:09.820968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:09.821415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:09.832676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:09.834591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:09.850320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:09.851530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:09.852653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:09.852653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:09.852653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:09.854466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 306/617] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:09.860016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:09.907761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 312/618] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:05:09.943982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:09.967100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18c857c-0076-4d86-92ff-768513e0c156 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:10.005808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:10.006837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.007272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.007784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.008584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:10.009062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:10.010642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:10.030315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 307/619] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:10.036190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e38052a-b65d-4760-9f0d-9dea8e13f189 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:10.101552 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:10.154132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e38052a-b65d-4760-9f0d-9dea8e13f189 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:10.203530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:10.204181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.204415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.204679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.212483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:10.213935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:10.225177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:10.226121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:10.227028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 313/620] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:10.231974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:10.322541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:10.402245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6760e06-9746-42b5-9b19-6c7b2416af02 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:10.461743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25e2f8-9a4c-42e3-8c2b-a2a29b8db99b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:10.477074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:10.477282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:10.477710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.477996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.479721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.479721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:10.479721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:10.481398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:10.494772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:10.508638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 308/621] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:10.512839 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:10.531196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:10.531838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.532120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.532393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.549969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:10.555254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:10.556190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:10.557091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:10.558677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 314/622] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:10.562755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:10.683191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7d127-ef26-40d9-927f-4005d0cdca71 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:10.710665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a54c4d-5f84-47f6-a200-511ff866ad4e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:10.727064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:10.727676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.727957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.728234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.728905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:10.729262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:10.730627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:10.746475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 315/623] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:10.753250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:10.779736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4938ad8-8f12-4a66-8077-e5e0f2a2913e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:10.780587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4938ad8-8f12-4a66-8077-e5e0f2a2913e None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.780963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4938ad8-8f12-4a66-8077-e5e0f2a2913e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.781248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4938ad8-8f12-4a66-8077-e5e0f2a2913e None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/3d69adfda7334567b03627a1d839f4f2/roles/8f8d9a3eff6c47faa94d854a3f42be35` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.805194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4938ad8-8f12-4a66-8077-e5e0f2a2913e None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=3d69adfda7334567b03627a1d839f4f2, role_id=8f8d9a3eff6c47faa94d854a3f42be35)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:10.806700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4938ad8-8f12-4a66-8077-e5e0f2a2913e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:10.822183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:10.831361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 309/624] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:10 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/3d69adfda7334567b03627a1d839f4f2/roles/8f8d9a3eff6c47faa94d854a3f42be35 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:05:10.911072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd593a-773d-4b82-a8ad-6f7d2c3b3e41 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:10.920939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69584646-cef5-40e8-a27d-0a6e49f1d398 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.921259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69584646-cef5-40e8-a27d-0a6e49f1d398 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.921503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69584646-cef5-40e8-a27d-0a6e49f1d398 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.923140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 310/625] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:10.931262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c28c982-3646-42c0-a1a3-a3e42c617794 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.931902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c28c982-3646-42c0-a1a3-a3e42c617794 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.932396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c28c982-3646-42c0-a1a3-a3e42c617794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.951032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:10.951600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:10.951949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:10.952227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:10.969049 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:10.970596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:10.981301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4c28c982-3646-42c0-a1a3-a3e42c617794 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:10.982937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4c28c982-3646-42c0-a1a3-a3e42c617794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:10.988954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:10.990531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:10.991616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:10.991616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:10.991616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:10.991616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 316/626] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:10.994560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:11.076655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:11.105362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 311/627] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:11.128139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:11.148558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd86f1fc-4308-4009-b947-973b8be548bb None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:11.196296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:11.197118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.197635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.198122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.199398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:11.199955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:11.201562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:11.215224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 317/628] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:11.220244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c28c982-3646-42c0-a1a3-a3e42c617794 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:11.358357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c28c982-3646-42c0-a1a3-a3e42c617794 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:11.406129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:11.406906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.407068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.407287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.412319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:11.415725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:11.417316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:11.436660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:11.438956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:11.441237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 312/629] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:05:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:11.444951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:11.520193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:11.584023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093fe1cd-d87e-49d1-8dcc-351b90a76094 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:11.616813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09409b3f-0e7b-4d7d-b05c-71680c69f1cb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:11.663952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:11.665057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.665764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.666987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.668548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:11.669243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:11.671799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:11.684131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f692e5-95cc-4351-8f6c-115fbf09153a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:11.685130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f692e5-95cc-4351-8f6c-115fbf09153a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.685678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f692e5-95cc-4351-8f6c-115fbf09153a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.686157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f692e5-95cc-4351-8f6c-115fbf09153a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.706590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 313/630] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:05:11 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:11.707781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f692e5-95cc-4351-8f6c-115fbf09153a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:11.713033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-e4f692e5-95cc-4351-8f6c-115fbf09153a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:11.715280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:11.716222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:11.717131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:11.720722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 318/631] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:11.722852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7b4e4-3b33-4cde-9104-9394bbcd77f8 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.722852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7b4e4-3b33-4cde-9104-9394bbcd77f8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.722852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d7b4e4-3b33-4cde-9104-9394bbcd77f8 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.724092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 319/632] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:11.728691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d7b4e4-3b33-4cde-9104-9394bbcd77f8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:11.732102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:11.832032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:11.839979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aca8e1-1287-480f-aa67-b18327ffa79f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:11.840152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d7b4e4-3b33-4cde-9104-9394bbcd77f8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:11.878338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a08da7-4e7b-4d42-931c-5b7cbf4b3c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:11.878938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a08da7-4e7b-4d42-931c-5b7cbf4b3c68 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.879217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a08da7-4e7b-4d42-931c-5b7cbf4b3c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.879491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a08da7-4e7b-4d42-931c-5b7cbf4b3c68 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.880150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a08da7-4e7b-4d42-931c-5b7cbf4b3c68 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:11.881114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a08da7-4e7b-4d42-931c-5b7cbf4b3c68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:11.882426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a08da7-4e7b-4d42-931c-5b7cbf4b3c68 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:11.882981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2efc33-7cba-4fcb-9718-78998c807210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:11.883816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2efc33-7cba-4fcb-9718-78998c807210 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.884315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2efc33-7cba-4fcb-9718-78998c807210 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.884777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2efc33-7cba-4fcb-9718-78998c807210 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.891730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 320/633] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:11.892947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14490075-1ca6-4760-accd-4fecbf640a04 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.893238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14490075-1ca6-4760-accd-4fecbf640a04 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.893513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14490075-1ca6-4760-accd-4fecbf640a04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.913680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-14490075-1ca6-4760-accd-4fecbf640a04 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:11.914871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-14490075-1ca6-4760-accd-4fecbf640a04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:11.965972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2efc33-7cba-4fcb-9718-78998c807210 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:11.968082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2efc33-7cba-4fcb-9718-78998c807210 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:11.977145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:11.986993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 321/634] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:11.988511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663a03e1-704a-4308-8548-3b8275098e76 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:11.988793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663a03e1-704a-4308-8548-3b8275098e76 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:11.989091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663a03e1-704a-4308-8548-3b8275098e76 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:11.990725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 322/635] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:11.991032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663a03e1-704a-4308-8548-3b8275098e76 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.086154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:12.093639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 314/636] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 01 04:05:12.095147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ec8728-a3bd-4ca7-ab85-03eaf4afb2ed None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.095838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ec8728-a3bd-4ca7-ab85-03eaf4afb2ed None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.096826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ec8728-a3bd-4ca7-ab85-03eaf4afb2ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.116457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-39ec8728-a3bd-4ca7-ab85-03eaf4afb2ed None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:12.117737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-39ec8728-a3bd-4ca7-ab85-03eaf4afb2ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:12.123786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663a03e1-704a-4308-8548-3b8275098e76 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.161131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.162224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.162711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.163185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.174362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:12.176167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:12.192673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:12.193889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:12.194791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:12.194791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:12.194791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:12.195717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 315/637] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:12.196715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ec8728-a3bd-4ca7-ab85-03eaf4afb2ed None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.197137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 323/638] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:12.206642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.257301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:12.318410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39101d1e-d28d-4792-a9cd-a63d6ffb72e6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.343369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ec8728-a3bd-4ca7-ab85-03eaf4afb2ed None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.356254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.357462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.357895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.358340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.359181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:12.359569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:12.361194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:12.385084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 324/639] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:12.386620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.390073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:12.408140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.410053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.410947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.411545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.426378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:12.429496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:12.448562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:12.449604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:12.450591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:12.450591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:12.450591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:12.450591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 316/640] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:12.450591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.520623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d85e94-20a1-4b06-9d6e-be2907fc150a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.530183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:12.565235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.565868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.566151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.566423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.577395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:12.582126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:12.583140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 325/641] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:12.584325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.593484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e37842e-c2db-4498-8af6-f91732fbfa6f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.628394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.629069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.629352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.629626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] PATH_INFO: `/v3/users/glance` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.637845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:12.639285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:12.651780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:12.652795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 317/642] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/users/glance => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:12.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.694549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:12.697405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8402d1-ca46-496d-a4d8-3cd7c62493b1 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.734590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.735201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.735485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.735758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.736407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:12.736711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:12.738137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:12.754700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 326/643] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:12.754700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.766602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0133efe7-27b9-40ee-a8db-d65feb6b292c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.781241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:12.803315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.803950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.804230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.804498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.814822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:12.819567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:12.820693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:12.821777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 318/644] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:12.826072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.866702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d59ecc-b9d0-4289-a541-642f47784505 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.903215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.903827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.904106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.904384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.905498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:12.905810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:12.907533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:12.917137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 327/645] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:12.918758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:12.926863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:12.931482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cbac3b-1875-4a2a-a5bc-5b1aec89c1db None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:12.968499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:12.969143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:12.969427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:12.969701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:12.970549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:12.971002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:12.972439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:12.996833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 319/646] 10.222.0.26 () {56 vars in 1137 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:12.997731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.012018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.025029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b64674-be01-4aa3-935d-48236cabca31 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.062451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.063169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.063452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.063725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.071780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.073221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:13.086712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:13.087598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:13.088514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 328/647] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:13.089041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.108537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b160ab-93a7-4a0d-91a7-00255dfb355d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.145185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.145726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.146057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.146332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.147014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:13.147314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.148718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.163039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 320/648] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:13.163039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.182583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.223566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8303c79b-f1ca-462a-af08-99ef8e76faf3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.260482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.261156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.261397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.261669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.264752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee9425d-c590-4157-9be0-b2dbf62d8508 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.271872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.273320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.281017 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:13.291457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:13.292565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:13.293590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:13.293590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:13.293590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:13.293590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 329/649] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:13.294055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.303737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e23a580-a0ae-4300-8c54-0eae8e95e7c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.304367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e23a580-a0ae-4300-8c54-0eae8e95e7c0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.304646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e23a580-a0ae-4300-8c54-0eae8e95e7c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.304933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e23a580-a0ae-4300-8c54-0eae8e95e7c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.369513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e23a580-a0ae-4300-8c54-0eae8e95e7c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.370963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e23a580-a0ae-4300-8c54-0eae8e95e7c0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.387587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.392378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dddd9f6-c782-4243-a2d5-a705df852df3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.427220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.428353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.428628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.428907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.429620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:13.429929 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.431351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.443047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 330/650] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:13.443838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.473638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 321/651] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:05:13.475636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59d23b1-ca13-49ab-ae5e-2becbaaffc51 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.475926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59d23b1-ca13-49ab-ae5e-2becbaaffc51 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.476210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59d23b1-ca13-49ab-ae5e-2becbaaffc51 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.477488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 322/652] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:13.481727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59d23b1-ca13-49ab-ae5e-2becbaaffc51 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.504753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.547786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c2b543-91bb-428f-9d93-44da47b07a79 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.582447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.583071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.583347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.584347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.593302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59d23b1-ca13-49ab-ae5e-2becbaaffc51 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.595162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.596594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.611305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:13.611713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:13.612640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:13.613577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:13.613577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:13.613577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:13.613577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 331/653] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:13.613806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.629170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.629774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.630051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.630354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.630996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:13.631295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.632629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.643698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 323/654] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:13.644471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.723207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3651e189-ca73-4a55-ade0-68d0288e76cc None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.735424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.742156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b0e88d-a793-4b94-8ba3-4c8d441cb893 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.756767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe514b-83bc-4caa-bd62-70329d3fae82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.757402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe514b-83bc-4caa-bd62-70329d3fae82 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.757732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe514b-83bc-4caa-bd62-70329d3fae82 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.758011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe514b-83bc-4caa-bd62-70329d3fae82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.758741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe514b-83bc-4caa-bd62-70329d3fae82 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=901b9bb47c2f40c2b1b4cfe7886a1167) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:13.759051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe514b-83bc-4caa-bd62-70329d3fae82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.760465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe514b-83bc-4caa-bd62-70329d3fae82 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.766128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 332/655] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/role_assignments?user.id=901b9bb47c2f40c2b1b4cfe7886a1167&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:13.767256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27510d-df97-4226-97ac-d935321a26cc None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.767417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27510d-df97-4226-97ac-d935321a26cc None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.767744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba27510d-df97-4226-97ac-d935321a26cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.775526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.776414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.776786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.777188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.790266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.790692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ba27510d-df97-4226-97ac-d935321a26cc None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:13.791914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ba27510d-df97-4226-97ac-d935321a26cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:13.792109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:13.811031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:13.812216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:13.818279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:13.818279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:13.818279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:13.818279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 324/656] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:13.818279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.852357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:13.877556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 333/657] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:13.878684 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba27510d-df97-4226-97ac-d935321a26cc None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:13.946602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b5d98-009e-4abf-82db-66c4045cdb0a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:13.990484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:13.991169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:13.991557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:13.991959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:13.992700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:13.993148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:13.994604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.008184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 325/658] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:14.009210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.029699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba27510d-df97-4226-97ac-d935321a26cc None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.040110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.080291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.081787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.081787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.081787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.097299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.100049 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:14.116550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:14.117744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 01 04:05:14.118920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:14.119598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.119685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 334/659] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:13 2022] GET /identity/v3/users/neutron => generated 85 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:14.120099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.134259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69cb0f4-3800-4ea4-9c03-c59740e5b9b9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.178496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.179221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.179575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.179917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.180625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:14.181030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.182494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.194245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 326/660] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:14.195047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.239237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c056321-dad4-4f22-8331-7fef5d3cb358 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.281896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.282821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.283109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.283800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.293088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.294522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:14.306420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:14.307381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:05:14.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:14.308914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4fad64-d180-420a-81fd-e94f1cfadb0d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.312021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 335/661] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/users/cinder => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:14.312808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.342231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.342576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.342837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.343114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.343382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.344108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:14.344526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.345874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.363887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 327/662] 10.222.0.26 () {56 vars in 1139 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:14.364828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.376473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.431826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975e6e0e-a0a3-4cf3-8f29-6c317e170b7c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.466744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.467467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.467818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.468271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.473783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff17527-3d73-4308-b757-74c2e5efc395 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.478592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:14.483757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:14.484778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 336/663] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:14.485649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.517667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.520466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.520754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.521589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.528975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.530695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:14.542915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:14.543814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:14.544699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:14.548832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 328/664] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:14.549582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.585247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.602341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44791c73-2ebf-463b-9c93-8d7e13d317b2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.617945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.640086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.640803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.641109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.641383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.642110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:14.642413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.643983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.652836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5f1e7-67a9-4af7-a5ce-ca8be93b7970 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.667407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 337/665] 10.222.0.26 () {56 vars in 1137 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:14.667407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.692060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.692660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.693667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.694024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.766122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.767604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.774771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a047a7a-c432-483a-badc-93c8bc7b6a9a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.814199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.814719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.815011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.815272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.815931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:14.816228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.817610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:14.826212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 338/666] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:14.827008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.840434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.845714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:14.886657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 329/667] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:05:14.886657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:14.948449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecb3f7-f9e4-44f7-a015-514a17224028 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.987192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:14.987794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:14.988075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:14.988350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:14.989069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:14.989383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:14.990591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4371b095-55be-424f-927f-b0e57d64e92f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:14.991365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.003605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 339/668] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:15.004394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.034211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.035155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.035782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.036379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.075380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.078543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.111971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b140e5c-6b27-46f4-b7e9-b4de50fad47d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.119260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.120709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.147613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.148404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.148783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.149187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.159803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.161396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:15.178055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:15.179168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:15.180046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:15.180046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:15.180046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:15.181637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 340/669] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:15.182963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.231410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 330/670] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:05:15.231410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.293794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.294833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207467cd-0e85-4ac0-b676-dac37442a996 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.314370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.327792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f570203-a9b8-46f1-9964-8314382d1ac0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.334827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.335853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.336139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.336412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.351244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.352667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.361075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.361790 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.362040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.362233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.362942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=170fe0394f3d4472a1380481cfe2cc28) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:15.363250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.365180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:15.367458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:15.368392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:15.369353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:15.369353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:15.369353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:15.369353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 341/671] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:15.369594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.381755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 331/672] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/role_assignments?user.id=170fe0394f3d4472a1380481cfe2cc28&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:15.381755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.499911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab66fb-1561-4104-a240-229ac833d584 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.519338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075b2b7e-1dc0-4b34-b7fa-ffc872b51666 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.548053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.549117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.549676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.549952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.550610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:15.550914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.552365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.555230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.561857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.562500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.562777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.563044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.567061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 342/673] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:15.568503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.581546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.582945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:15.600271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:15.601341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:15.602254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:15.602254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:15.602254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:15.602254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 332/674] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:15.602519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.607297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.692639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa7f28-c301-4bdc-8b1d-8fb0a81c6b93 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.733302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cdf9c2-1118-4af8-b8a8-335a3e74fd5b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.745265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.746285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.746741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.747429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.748724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:15.749559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.751716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.768640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 343/675] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:15.769745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.777318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.778031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.778372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.778712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] PATH_INFO: `/v3/users/swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.790215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.792067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:15.808864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:15.809906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:15.810842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:15.814850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 333/676] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/users/swift => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:15.816840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.879136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.892292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:15.906166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d480881-f4da-424a-a81a-79c63472c72e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.946472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.947531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.948016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.948480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:15.949583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:15.950500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:15.952326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:15.952827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52662c6-0506-4b9d-9f71-98cd6e4e698a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:15.969877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 344/677] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:15.971035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:15.994489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:15.995683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:15.996163 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:15.996631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.004569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.006494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:16.019618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:16.020530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:16.024797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 334/678] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:16.025867 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.098545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79975f82-fffa-448a-8129-2b824e2a7428 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.131538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1104789a-6643-4a94-85bf-f7ce8c39945a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.133817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:16.136524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.137311 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.137513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.137781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.138640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:16.139400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.140753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.164638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 345/679] 10.222.0.26 () {56 vars in 1135 bytes} [Sat Oct 1 04:05:15 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:16.164638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.173595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:16.174223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3277c97-8507-492e-9756-bedc5126c94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.174864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3277c97-8507-492e-9756-bedc5126c94b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.175164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3277c97-8507-492e-9756-bedc5126c94b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.175413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3277c97-8507-492e-9756-bedc5126c94b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.182953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3277c97-8507-492e-9756-bedc5126c94b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.184362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3277c97-8507-492e-9756-bedc5126c94b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c3277c97-8507-492e-9756-bedc5126c94b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:16.195600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:16.196501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:16.197455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 335/680] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:16.198634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f607a2-6f56-4ce9-8134-e1873332936c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.198634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f607a2-6f56-4ce9-8134-e1873332936c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.198634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f607a2-6f56-4ce9-8134-e1873332936c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.199763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 336/681] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:16 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:16.204319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f607a2-6f56-4ce9-8134-e1873332936c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.274824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc43550-c27f-4648-a012-6038c623ac18 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.309601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.310495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.311010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.311510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.312556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:16.312871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.314339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.315731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f607a2-6f56-4ce9-8134-e1873332936c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.328293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 346/682] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:16.329152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.358057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.359122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.359625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.360091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.395789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:16.405921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:16.438541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.440268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.446363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e370fb1b-3ce1-4e24-9a78-efd9e312e3c5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.483233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bcfbdc-52a2-4047-8795-3bb7c78684fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.484088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bcfbdc-52a2-4047-8795-3bb7c78684fe None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.484512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bcfbdc-52a2-4047-8795-3bb7c78684fe None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.484958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bcfbdc-52a2-4047-8795-3bb7c78684fe None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.485799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bcfbdc-52a2-4047-8795-3bb7c78684fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:16.486110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bcfbdc-52a2-4047-8795-3bb7c78684fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.487526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35bcfbdc-52a2-4047-8795-3bb7c78684fe None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.507292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 347/683] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:16.508627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fc78d9-91aa-4cfb-8cea-783de5d4aa7c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.508911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fc78d9-91aa-4cfb-8cea-783de5d4aa7c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.509202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fc78d9-91aa-4cfb-8cea-783de5d4aa7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.528770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-98fc78d9-91aa-4cfb-8cea-783de5d4aa7c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:16.530004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-98fc78d9-91aa-4cfb-8cea-783de5d4aa7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:16.549425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 337/684] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:05:16.550270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.606686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 348/685] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:16.610088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fc78d9-91aa-4cfb-8cea-783de5d4aa7c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.617722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:16.658164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5400561-48e3-4e9f-ab5c-f0118b324517 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.694924 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.695507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.695793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.696062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.696960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=16aab142c0a7445483caf19178d56062) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:16.697139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.698574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.699611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:16.705905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 338/686] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/role_assignments?user.id=16aab142c0a7445483caf19178d56062&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:16.706796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.758111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fc78d9-91aa-4cfb-8cea-783de5d4aa7c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.837003 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.839683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.840221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.841078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.842742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=52b5d64c999a4f16afec678df3025529) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:16.846619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.848388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.861501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 349/687] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/role_assignments?user.id=52b5d64c999a4f16afec678df3025529&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 608 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:16.871327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18539997-777e-452a-90a8-4c1807e0a19c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:16.904319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:16.917917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:16.918944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:16.919432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:16.919798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:16.932473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:16.934156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:16.949559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:16.950768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:16.951829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:16.951829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:16.951829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:16.952047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:16.953033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 339/688] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:16.960441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.089494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3098214-2b82-4fd4-ad17-2dfa07fc9149 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.090897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d6b75-574c-4c31-9ca4-09c3f0a53f84 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.119783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:17.134707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.135521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.135876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.136156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.138957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.139573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.139900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.140182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.140835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:17.141175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:17.142548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:17.156155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:17.159997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 340/689] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:17.164749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:17.165729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:17.166568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:17.167211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.169556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 350/690] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:17.175204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.218947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:17.289615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b07ef2-a00a-4fb5-ad36-4ae7afe16599 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.301297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4b506-b817-4bfe-a801-cfcf17dcf911 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.336868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.337536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.337809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.338309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.339104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:17.339514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:17.341007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:17.345890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.346826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.347242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.347655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.351321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 351/691] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:17.356879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.358864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:17.360721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:17.384120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:17.385366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:17.386453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:17.395063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 341/692] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:17.401012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.418785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:17.481992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c22c5-df69-491d-99d8-fcd4848cf411 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.545685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.546387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.546676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.546955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.569128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:17.572738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbca451-c4c7-4e6b-8bf3-9660c9e1ed94 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.578190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:17.579666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:17.607246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:17.608408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:17.609491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:17.609491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:17.609491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:17.609491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 352/693] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:17.611037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.611708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.612047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.612378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.613235 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:17.613544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:17.614177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.615509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:17.630248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 342/694] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:17.635963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.801837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d12a0f2-cd8b-4398-83ae-f1d7540bd993 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.805283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:17.815467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d3e89a-0d06-4d87-a7af-0a4497e54ccf None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:17.868260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1f87e1-d6c6-471a-91cd-81831d3d8ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.868869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1f87e1-d6c6-471a-91cd-81831d3d8ff2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.869167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1f87e1-d6c6-471a-91cd-81831d3d8ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.869436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1f87e1-d6c6-471a-91cd-81831d3d8ff2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.870165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1f87e1-d6c6-471a-91cd-81831d3d8ff2 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=3d69adfda7334567b03627a1d839f4f2) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:17.870469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1f87e1-d6c6-471a-91cd-81831d3d8ff2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:17.871893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1f87e1-d6c6-471a-91cd-81831d3d8ff2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:17.881112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:17.881807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.882095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.882370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.882611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 343/695] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:17 2022] GET /identity/v3/role_assignments?user.id=3d69adfda7334567b03627a1d839f4f2&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 608 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:17.883132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:17.883444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:17.883841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d26709b-0474-4ba4-87f0-9e4d9f13c354 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.884126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d26709b-0474-4ba4-87f0-9e4d9f13c354 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.884394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d26709b-0474-4ba4-87f0-9e4d9f13c354 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.884947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:17.893948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 344/696] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:17 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:17.901801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f72ef-8e7e-4c61-aa65-9ae380ab6f88 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:17.902116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f72ef-8e7e-4c61-aa65-9ae380ab6f88 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:17.902355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f72ef-8e7e-4c61-aa65-9ae380ab6f88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:17.909012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 353/697] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:17.915104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:17.939405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-870f72ef-8e7e-4c61-aa65-9ae380ab6f88 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:17.940637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-870f72ef-8e7e-4c61-aa65-9ae380ab6f88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:17.977194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:18.057430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 345/698] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:05:18.094760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d2115-6f9b-4e5d-b02c-da44b742041e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:18.131816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:18.132450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:18.132724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:18.133014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:18.140414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:18.141933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:18.153691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:18.154648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:18.155593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 354/699] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:05:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:18.162208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870f72ef-8e7e-4c61-aa65-9ae380ab6f88 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:18.269921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870f72ef-8e7e-4c61-aa65-9ae380ab6f88 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:18.307859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:18.308385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:18.308665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:18.308949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:18.309673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:18.309974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:18.311378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:18.329969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 346/700] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:05:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:18.335891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:18.355796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:18.380572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:18.395650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:18.450031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a3d71f-3b2c-4edb-bbca-142be9b59d50 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:18.494373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:18.494985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:18.495265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:18.495537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:18.505798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fd067f-0a2e-4fd5-a66b-3ba6683421c9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:18.551273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:18.551926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:18.552294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:18.552583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:18.564241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:18.565784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:18.576783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:18.578375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:18.583016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:18.584642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:18.586257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:18.586257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:18.586257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:18.589058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 347/701] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:18.590866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:18.625295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:18.693729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:18.725633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 355/702] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 01 04:05:18.731018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:18.785342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b95fb79-555c-4e7c-8390-df0e90c21cc7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:18.850162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:18.851722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:18.852009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:18.852284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:18.853169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:18.853479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:18.854847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:18.881509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 348/703] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:18.886121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:18.914685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2ea2ea-b07b-49c1-bc3a-591a06c8d62c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:18.949311 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:18.966718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:18.967371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:18.967653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:18.967939 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:18.979435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:18.981041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:18.998031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:18.999229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:19.000475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:19.000475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:19.000475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:19.000475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 356/704] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:19.005195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:19.100732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee95c46-483e-4cd8-8d94-9af9e86b1bfa None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:19.148797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:19.161446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf096cef-cd80-4075-8271-a9a14b3af401 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:19.174133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:19.174754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.175031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.175304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.175973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:19.177473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:19.220811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:19.223305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.223605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.228093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.228735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 349/705] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:05:18 2022] POST /identity/v3/users => generated 254 bytes in 342 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:05:19.228886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:19.229198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:19.230627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:19.248414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 357/706] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:19.251026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:19.398783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b25aa7-2bdb-4a3f-9a51-fbf369fb07e4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:19.444751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcca8085-2952-4d09-b676-9ddfa320660f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:19.445795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcca8085-2952-4d09-b676-9ddfa320660f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.446280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcca8085-2952-4d09-b676-9ddfa320660f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.446754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcca8085-2952-4d09-b676-9ddfa320660f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.454098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcca8085-2952-4d09-b676-9ddfa320660f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:19.455764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcca8085-2952-4d09-b676-9ddfa320660f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-bcca8085-2952-4d09-b676-9ddfa320660f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:19.468174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:19.469104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:19.470000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:19.472385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 350/707] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:19.478159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:19.516872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:19.600079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5d65b9-d9c8-4ca0-ad36-e78403d250b5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:19.641604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:19.642225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.642503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.642780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.643551 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:19.643861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:19.645339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:19.649125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fa6690-cd89-435e-bbdd-4d55db317915 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.650084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fa6690-cd89-435e-bbdd-4d55db317915 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.650576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fa6690-cd89-435e-bbdd-4d55db317915 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.652457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 351/708] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:19.657942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4cdc3-d073-4a10-a72f-c0c0574b0950 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.658760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4cdc3-d073-4a10-a72f-c0c0574b0950 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.659252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4cdc3-d073-4a10-a72f-c0c0574b0950 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.668953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 358/709] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:19.672592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:19.684583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-13b4cdc3-d073-4a10-a72f-c0c0574b0950 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:19.686557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-13b4cdc3-d073-4a10-a72f-c0c0574b0950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:19.727010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:19.772688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 352/710] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:05:19.774479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5b0ab5-2a51-4399-bcd7-ff107cd36e1c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.774996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5b0ab5-2a51-4399-bcd7-ff107cd36e1c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.775473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5b0ab5-2a51-4399-bcd7-ff107cd36e1c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.777196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 353/711] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:19.789206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:19.790677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cfe067-00c1-4054-82a1-ba13421bc4c3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.790964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cfe067-00c1-4054-82a1-ba13421bc4c3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.791277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cfe067-00c1-4054-82a1-ba13421bc4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.804568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54d8e4-3f23-4ad6-bf4c-3a70422d5884 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:19.840642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:19.841673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:19.841916 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b7cfe067-00c1-4054-82a1-ba13421bc4c3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:19.842397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:19.842874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:19.843102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b7cfe067-00c1-4054-82a1-ba13421bc4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:19.844055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=901b9bb47c2f40c2b1b4cfe7886a1167) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:19.844599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:19.846346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:19.853087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 359/712] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:19 2022] GET /identity/v3/role_assignments?user.id=901b9bb47c2f40c2b1b4cfe7886a1167&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:19.974914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 354/713] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:05:20.166693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:20.297395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b24932e-3b37-4f96-8560-dd1ac3823135 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:20.333187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:20.333963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.334249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.334537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.335120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:20.336477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:20.341715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cfe067-00c1-4054-82a1-ba13421bc4c3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:20.348742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:20.350536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:20.351477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 360/714] 10.222.0.26 () {56 vars in 1129 bytes} [Sat Oct 1 04:05:20 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:20.364067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:20.390640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:20.470687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dbe552-dde5-4861-b788-8d891ac68f66 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:20.513114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:20.513750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.514030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.514321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.526531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:20.526859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:20.528250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:20.533179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 361/715] 10.222.0.26 () {56 vars in 1124 bytes} [Sat Oct 1 04:05:20 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:20.534622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cfe067-00c1-4054-82a1-ba13421bc4c3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:20.537282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:20.540544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:20.572681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:20.573369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.573651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.573919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] PATH_INFO: `/v3/services/network` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.574505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:20.575845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:20.588134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:20.589166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:20.590594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 355/716] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:20 2022] GET /identity/v3/services/network => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:20.594968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:20.704772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f01d3-815b-4f45-82df-db72c8d8ee36 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:20.743158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:20.743782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.744067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.744346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.745003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] RBAC: Adding query filter params (name=network) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:20.745316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:20.746678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:20.751894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 356/717] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:20 2022] GET /identity/v3/services?name=network => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:20.758223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:20.792149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:20.850059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:20.892950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64ef3a9-d374-4be5-bd27-9d25c3239906 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:20.910188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4abe82-dd47-427a-9670-f2c3297819f2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:20.950984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba6093f-6522-425f-87a0-fa25953c56b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:20.952005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6093f-6522-425f-87a0-fa25953c56b1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.952527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6093f-6522-425f-87a0-fa25953c56b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.952820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6093f-6522-425f-87a0-fa25953c56b1 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.953772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba6093f-6522-425f-87a0-fa25953c56b1 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:20.954081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba6093f-6522-425f-87a0-fa25953c56b1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:20.956113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba6093f-6522-425f-87a0-fa25953c56b1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:20.959527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:20.960145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.960430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.960707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.961371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] RBAC: Adding query filter params (type=network) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:20.961683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:20.962173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 362/718] 10.222.0.26 () {56 vars in 1124 bytes} [Sat Oct 1 04:05:20 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 422 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:20.963041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:20.963430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f931c09-a566-4eb3-affe-68d71a0e7ada None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.963718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f931c09-a566-4eb3-affe-68d71a0e7ada None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.964045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f931c09-a566-4eb3-affe-68d71a0e7ada None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.970499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 363/719] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:20.971967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f265c50a-239d-4e45-9763-c22d59941353 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:20.972255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f265c50a-239d-4e45-9763-c22d59941353 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:20.972529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f265c50a-239d-4e45-9763-c22d59941353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:20.980643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 357/720] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:20 2022] GET /identity/v3/services?type=network => generated 124 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:20.993057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f265c50a-239d-4e45-9763-c22d59941353 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:20.994338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f265c50a-239d-4e45-9763-c22d59941353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:21.075215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 364/721] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:21.347106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:21.444322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:21.453890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09621e9c-efc8-4fff-bad4-33560175af2e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:21.487778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:21.488406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:21.488685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:21.488977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:21.499880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:21.505562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:21.506800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:21.508277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 358/722] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:21.512713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f265c50a-239d-4e45-9763-c22d59941353 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:21.589109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:21.614517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f265c50a-239d-4e45-9763-c22d59941353 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:21.649321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:21.649932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:21.650210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:21.650478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:21.651141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:21.651441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:21.652789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:21.662222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 365/723] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:21.667801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:21.769330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeeee9e-77d3-40fe-ae50-d8cc663e6cdd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:21.804143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:21.804781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:21.805357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:21.805728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:21.817314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:21.819345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:21.834699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:21.835656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:21.837248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:21.837248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:21.837248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:21.837248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 359/724] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:21.841694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:21.944221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d05b6b-ac43-42c8-9b7b-3c619de300ca None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:21.978824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:21.979459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:21.979703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:21.979972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:21.980610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:21.980912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:21.982297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:21.984704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:22.000752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:22.001218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 366/725] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:22.005368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:22.107982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e833ef-280f-4b7b-80c0-bcf17cde09ac None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:22.146045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:22.146862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:22.147146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:22.147415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:22.154971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:22.156412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:22.168443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:22.169498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:22.170493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 360/726] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:21 2022] GET /identity/v3/users/swift => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:22.175490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:22.283135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e62d5-9812-4a74-87af-caed7244a522 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:22.319264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77198c1-40e3-45f6-a319-7b1077102bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:22.319869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77198c1-40e3-45f6-a319-7b1077102bea None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:22.320145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77198c1-40e3-45f6-a319-7b1077102bea None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:22.320413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77198c1-40e3-45f6-a319-7b1077102bea None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:22.321161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77198c1-40e3-45f6-a319-7b1077102bea None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:22.321462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77198c1-40e3-45f6-a319-7b1077102bea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:22.322889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77198c1-40e3-45f6-a319-7b1077102bea None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:22.340754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 367/727] 10.222.0.26 () {56 vars in 1135 bytes} [Sat Oct 1 04:05:22 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:22.346405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:22.385073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5361642-1459-427d-ac86-2d76c152ce1e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:22.385396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5361642-1459-427d-ac86-2d76c152ce1e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:22.385622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5361642-1459-427d-ac86-2d76c152ce1e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:22.387195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 368/728] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:22.394605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64517db1-64ab-4bfa-a729-65484d0fe3ea None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:22.395019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64517db1-64ab-4bfa-a729-65484d0fe3ea None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:22.395395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64517db1-64ab-4bfa-a729-65484d0fe3ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:22.418352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-64517db1-64ab-4bfa-a729-65484d0fe3ea None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:22.419555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-64517db1-64ab-4bfa-a729-65484d0fe3ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:22.453375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:22.493300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:22.499952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 369/729] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:22.564711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520b91c7-48d6-4cb9-86c0-a44c88c5ea5b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:22.582143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba4a9fb-15f5-44de-83e0-355042a66872 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:22.582753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba4a9fb-15f5-44de-83e0-355042a66872 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:22.583270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba4a9fb-15f5-44de-83e0-355042a66872 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:22.585114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 370/730] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:22.590227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8883df86-96f2-4e6b-bb74-a383e75de872 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:22.591185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8883df86-96f2-4e6b-bb74-a383e75de872 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:22.591673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8883df86-96f2-4e6b-bb74-a383e75de872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:22.622123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8883df86-96f2-4e6b-bb74-a383e75de872 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:22.623052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:22.623894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:22.624399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:22.624870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:22.627169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8883df86-96f2-4e6b-bb74-a383e75de872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:22.745827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:22.749653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:22.765428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 371/731] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 01 04:05:22.820243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8883df86-96f2-4e6b-bb74-a383e75de872 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:22.873298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:22.940654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 361/732] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 595 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 01 04:05:22.945940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:22.965556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8883df86-96f2-4e6b-bb74-a383e75de872 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:22.982040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:23.014619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.015586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.016094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.016574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.041524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:23.051262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:23.052262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:23.053161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:23.053812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 372/733] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:23.064414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.124834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3ed643-97bd-448e-9fa8-b4af99d73bc3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.168070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.169513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.170007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.170820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.184362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.186863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:23.194011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0b64b4-c46b-4377-80ce-5ee3c40b97ac None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:23.211525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:23.212747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:23.213890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:23.213890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:23.213890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:23.215562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 362/734] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:23.216487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.239993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.240522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:23.240841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.241451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.241571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.242503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:23.243247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.244788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:23.257007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 373/735] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:23.258078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.303097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:23.358028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09525835-ee22-4768-9009-ec171bbf6c54 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.379828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2890fae-aaa6-42bb-a57b-f81a329a1f4c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.404237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.404876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.405188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.405462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.422114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:23.426290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.427755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:23.428333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:23.429321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:23.430184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:23.430811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.431727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.432597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:23.433253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.434208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 363/736] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:23.435051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.435315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:23.457668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 374/737] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:23.458737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.545794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:23.577975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef71ca0-a92c-48e1-89f1-a4ad34bf1f6f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.579485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:23.587540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79143d5-2096-4f10-953a-2fddea4c5f9e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.619922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd2fbe0-b838-4c0c-9038-0eddf455a8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.620564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fbe0-b838-4c0c-9038-0eddf455a8a0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.620847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fbe0-b838-4c0c-9038-0eddf455a8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.621206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fbe0-b838-4c0c-9038-0eddf455a8a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.624066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.624682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.624986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.625266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.625936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:23.626242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.627600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:23.632669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd2fbe0-b838-4c0c-9038-0eddf455a8a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.634084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd2fbe0-b838-4c0c-9038-0eddf455a8a0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:23.636976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 375/738] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:23.637726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-4dd2fbe0-b838-4c0c-9038-0eddf455a8a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:23.652894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:23.653883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:23.654758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:23.654758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:23.654758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:23.654758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 364/739] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:23.656359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47634c86-0896-416a-ae76-6d0e1222d67e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.656359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47634c86-0896-416a-ae76-6d0e1222d67e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.656359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47634c86-0896-416a-ae76-6d0e1222d67e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.657211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 365/740] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:23 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:23.657921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47634c86-0896-416a-ae76-6d0e1222d67e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.757667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdebef32-1cd9-4281-9142-2b7bd669d37d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.772104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47634c86-0896-416a-ae76-6d0e1222d67e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.795238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4863e9-3119-4025-9b3a-895e34a05d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.795955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4863e9-3119-4025-9b3a-895e34a05d9f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.796253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4863e9-3119-4025-9b3a-895e34a05d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.796525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4863e9-3119-4025-9b3a-895e34a05d9f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.800742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:23.805559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4863e9-3119-4025-9b3a-895e34a05d9f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.806981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4863e9-3119-4025-9b3a-895e34a05d9f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:23.810060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.810702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.810978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.811318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-ad4863e9-3119-4025-9b3a-895e34a05d9f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:23.819755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:23.820753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:23.821720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 376/741] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:23.822213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9b568-85ff-47e0-8ef6-9d97186fd2ec None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.822455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9b568-85ff-47e0-8ef6-9d97186fd2ec None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.822715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9b568-85ff-47e0-8ef6-9d97186fd2ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.824454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.826064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:23.841095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-16b9b568-85ff-47e0-8ef6-9d97186fd2ec None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:23.842292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-16b9b568-85ff-47e0-8ef6-9d97186fd2ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:23.845775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:23.846871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:23.848091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:23.848091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:23.848091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:23.848091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 366/742] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:23.848091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.856039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:23.909898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 377/743] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:23.911063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7cd4f-4b20-4965-b41e-492c519a287c None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.911991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7cd4f-4b20-4965-b41e-492c519a287c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.912275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b7cd4f-4b20-4965-b41e-492c519a287c None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.918496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 378/744] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:23.918496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7cd4f-4b20-4965-b41e-492c519a287c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:23.954948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e380f29d-14c4-4ade-acdd-4c705f10ac39 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:23.992607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:23.993235 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:23.993508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:23.993773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:23.994399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:23.994694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:23.996079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.008322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 367/745] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:24.009476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.021196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b7cd4f-4b20-4965-b41e-492c519a287c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.060030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2bab39-e33e-42dc-8cbb-a6757cc21a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.061076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2bab39-e33e-42dc-8cbb-a6757cc21a96 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.061600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2bab39-e33e-42dc-8cbb-a6757cc21a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.062069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2bab39-e33e-42dc-8cbb-a6757cc21a96 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.063048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2bab39-e33e-42dc-8cbb-a6757cc21a96 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:24.063567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2bab39-e33e-42dc-8cbb-a6757cc21a96 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.065475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2bab39-e33e-42dc-8cbb-a6757cc21a96 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.083822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 379/746] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:24.085434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b78e44-1d69-417f-bbc5-730d42bd7267 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.085738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b78e44-1d69-417f-bbc5-730d42bd7267 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.086056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b78e44-1d69-417f-bbc5-730d42bd7267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.106989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-26b78e44-1d69-417f-bbc5-730d42bd7267 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:24.108206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-26b78e44-1d69-417f-bbc5-730d42bd7267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:24.110738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:24.171257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:24.185253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 380/747] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 01 04:05:24.186316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b78e44-1d69-417f-bbc5-730d42bd7267 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.237029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8781347b-e092-485e-88f4-ad9e43032be7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.272705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.273704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.273989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.274256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.275201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:24.275510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.276872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.292425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 368/748] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:24.292425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.305038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b78e44-1d69-417f-bbc5-730d42bd7267 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.342342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.343121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.343497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.343866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.351103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.352638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.354911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:24.367434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:24.368554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:24.369563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 381/749] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:05:24 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:24.370153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.399881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:24.403287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7589090e-34c2-491f-a857-bb13b1123666 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.438955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.439778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.440060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.440332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.441080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=3d69adfda7334567b03627a1d839f4f2) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:24.441400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.442877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.448966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 369/750] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:24 2022] GET /identity/v3/role_assignments?user.id=3d69adfda7334567b03627a1d839f4f2&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:24.449254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.496252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7782c8-2a3b-4f57-8c2c-e7ebb2b53e7c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.541775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.542866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.542866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.543040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.543583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.545261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.563689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 382/751] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:24 2022] POST /identity/v3/services => generated 251 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:24.564848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.585405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3839a373-ef9f-4099-b225-9e49a8e0396e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.636726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.637665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.637943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.638213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.647020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.648484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.649673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:24.663958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:24.665352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:24.666339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:24.669982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 370/752] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:05:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:24.670933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.695644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:24.732251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd6a75b-c638-44cf-8fda-fce079dea33c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.781608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.782230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.782514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.782790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.783781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:24.784094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.785587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.791482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb379499-7152-4116-9d0b-f0572af15a71 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.807789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 383/753] 10.222.0.26 () {56 vars in 1149 bytes} [Sat Oct 1 04:05:24 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:24.808801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.844771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:24.845436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:24.845767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:24.846293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:24.847884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:24.850060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:24.876091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 371/754] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:24 2022] POST /identity/v3/services => generated 240 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:24.877302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:24.950009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22377d8e-0bd8-40f3-b5ee-41acdc613868 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:24.968966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:25.005005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:25.006194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:25.007045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:25.007479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:25.007891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:25.008853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:25.009357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:25.010997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:25.034110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 384/755] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:05:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:25.042235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:25.058037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73c92ab-9999-43e2-bafc-a6e34d6e46fc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:25.115367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:25.115993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:25.116273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:25.116549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:25.209819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad0056e-b140-496e-8a8a-6de025ce80b9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:25.218085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:25.219542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:25.260547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:25.261227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:25.261504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:25.261771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:25.282886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:25.284585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:25.312413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:25.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:25.314430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:25.314430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:25.314430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:25.314430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 385/756] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:25.320397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:25.346434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:25.401394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:25.430869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12d077-28a8-4210-a4d6-8b2b4495aff2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:25.445447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 372/757] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:24 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 569 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:05:25.450720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:25.479916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:25.480456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:25.480731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:25.481042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:25.481693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:25.481993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:25.483547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:25.510962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 386/758] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:25.512123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:25.621239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:25.641890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44734e5b-c834-4e92-abca-5d897dd34a3c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:25.660637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903f51c8-ec2f-4d8d-b02b-a54d981b64ce None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:25.679243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:25.679882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:25.680162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:25.680434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:25.688136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:25.689704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:25.702811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:25.703803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:25.704708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 387/759] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:25.713945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:25.738831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9581cf39-db04-4040-8d61-917156251539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:25.739920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9581cf39-db04-4040-8d61-917156251539 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:25.740463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9581cf39-db04-4040-8d61-917156251539 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:25.740463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9581cf39-db04-4040-8d61-917156251539 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:25.764085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9581cf39-db04-4040-8d61-917156251539 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:25.765559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9581cf39-db04-4040-8d61-917156251539 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-9581cf39-db04-4040-8d61-917156251539 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:25.790124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:25.791122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:25.792016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:25.792016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:25.792016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:25.792016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 373/760] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:25.796657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9581cf39-db04-4040-8d61-917156251539 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:25.847758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:25.874996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fb5d1-b408-47c2-999b-26cc4363faa9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:25.909699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:25.910317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:25.910603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:25.910863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:25.911673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:25.912029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:25.913498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:25.933038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 388/761] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:25.935413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:25.938689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:25.977234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9581cf39-db04-4040-8d61-917156251539 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:26.015638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:26.016252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:26.016466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:26.016694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:26.017372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:26.017675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:26.019031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:26.033584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 374/762] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:26.042291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:26.124257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf15f72-93d2-4c5c-90d5-1970034aa32a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:26.162232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:26.163715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:26.164204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:26.164682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/901b9bb47c2f40c2b1b4cfe7886a1167/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:26.166945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=901b9bb47c2f40c2b1b4cfe7886a1167, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:26.168697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:26.185091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 389/763] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:25 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/901b9bb47c2f40c2b1b4cfe7886a1167/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:05:26.227872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:26.238319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4d98c9-9088-4da4-978f-588fa5faebc0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:26.275864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:26.276437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:26.276718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:26.277021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:26.284917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:26.287006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:26.299254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:26.300199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:26.301119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 375/764] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:26.306507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:26.413625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:26.456281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33369757-0257-4243-bee6-5a1b71f4f021 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:26.493628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c80287-cebe-4b15-b09e-dbc986db7b61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:26.494719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c80287-cebe-4b15-b09e-dbc986db7b61 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:26.495529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c80287-cebe-4b15-b09e-dbc986db7b61 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:26.496310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c80287-cebe-4b15-b09e-dbc986db7b61 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:26.497641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c80287-cebe-4b15-b09e-dbc986db7b61 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:26.498199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c80287-cebe-4b15-b09e-dbc986db7b61 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:26.499845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c80287-cebe-4b15-b09e-dbc986db7b61 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:26.527748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 390/765] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:26.533517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:26.670774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92e219e-585e-4399-a41d-27351c56be3b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:26.723546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31918801-09f1-4c7e-b1c6-2432a7abd723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:26.725094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31918801-09f1-4c7e-b1c6-2432a7abd723 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:26.725581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31918801-09f1-4c7e-b1c6-2432a7abd723 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:26.726102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31918801-09f1-4c7e-b1c6-2432a7abd723 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:26.727056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31918801-09f1-4c7e-b1c6-2432a7abd723 None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=98e0eece1e7146c48cf68b599e58c0bf) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:26.727582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31918801-09f1-4c7e-b1c6-2432a7abd723 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:26.729227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31918801-09f1-4c7e-b1c6-2432a7abd723 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:26.735622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 376/766] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:26 2022] GET /identity/v3/role_assignments?user.id=98e0eece1e7146c48cf68b599e58c0bf&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 259 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:26.971628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209d1cc6-2468-478c-a6d9-1f2b350d831b None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:26.972181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209d1cc6-2468-478c-a6d9-1f2b350d831b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:26.972181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209d1cc6-2468-478c-a6d9-1f2b350d831b None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:26.974343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 391/767] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:26.979690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d87ef-fd6a-4af6-8e39-c757ac0448f3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:26.979983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d87ef-fd6a-4af6-8e39-c757ac0448f3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:26.980258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d87ef-fd6a-4af6-8e39-c757ac0448f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.010227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1f9d87ef-fd6a-4af6-8e39-c757ac0448f3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:27.011438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1f9d87ef-fd6a-4af6-8e39-c757ac0448f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:27.026317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:27.106719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 377/768] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:26 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:27.202876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d77866-5a7e-46d5-9ed3-971051e55d61 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:27.203193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d77866-5a7e-46d5-9ed3-971051e55d61 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:27.203475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d77866-5a7e-46d5-9ed3-971051e55d61 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.205091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 392/769] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:27.211145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc1fe95-8e8b-4b8e-a86e-928f58a40f2a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:27.211436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc1fe95-8e8b-4b8e-a86e-928f58a40f2a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:27.211712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc1fe95-8e8b-4b8e-a86e-928f58a40f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.238504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fcc1fe95-8e8b-4b8e-a86e-928f58a40f2a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:27.239675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fcc1fe95-8e8b-4b8e-a86e-928f58a40f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:27.331640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 378/770] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:27 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:27.477661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d77866-5a7e-46d5-9ed3-971051e55d61 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:27.617723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d77866-5a7e-46d5-9ed3-971051e55d61 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:27.658734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:27.674493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:27.675134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:27.675412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:27.675681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.686331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:27.691024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:27.691979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:27.693412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 393/771] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:27.702104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc1fe95-8e8b-4b8e-a86e-928f58a40f2a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:27.705667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:27.787264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:27.812342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc1fe95-8e8b-4b8e-a86e-928f58a40f2a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:27.842537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f41f7c-bac1-4dc0-ad84-07b3c04ccdd2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:27.850848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546216d2-0588-48e2-8244-864066edbe0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:27.851747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546216d2-0588-48e2-8244-864066edbe0f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:27.852221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546216d2-0588-48e2-8244-864066edbe0f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:27.852687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546216d2-0588-48e2-8244-864066edbe0f None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.853571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546216d2-0588-48e2-8244-864066edbe0f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:27.854084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546216d2-0588-48e2-8244-864066edbe0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:27.855619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546216d2-0588-48e2-8244-864066edbe0f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:27.864501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 379/772] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:27.866071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f22ea5-e8df-4ff2-bf59-7260d2705862 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:27.866568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f22ea5-e8df-4ff2-bf59-7260d2705862 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:27.867058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f22ea5-e8df-4ff2-bf59-7260d2705862 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.868967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 380/773] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:27.869877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f22ea5-e8df-4ff2-bf59-7260d2705862 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:27.884503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d534c-e870-4426-ad42-de3f1f44e97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:27.885184 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d534c-e870-4426-ad42-de3f1f44e97c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:27.885474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d534c-e870-4426-ad42-de3f1f44e97c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:27.885754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d534c-e870-4426-ad42-de3f1f44e97c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.886353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512d534c-e870-4426-ad42-de3f1f44e97c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:27.887823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512d534c-e870-4426-ad42-de3f1f44e97c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-512d534c-e870-4426-ad42-de3f1f44e97c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:27.897554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:27.898550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:27.899396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 394/774] 10.222.0.26 () {56 vars in 1129 bytes} [Sat Oct 1 04:05:27 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:27.900186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f80749e-7402-4537-855a-a09d267b9851 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:27.905401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f80749e-7402-4537-855a-a09d267b9851 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:27.905401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f80749e-7402-4537-855a-a09d267b9851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:27.921805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-6f80749e-7402-4537-855a-a09d267b9851 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:27.923337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6f80749e-7402-4537-855a-a09d267b9851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:27.982731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:27.989096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f22ea5-e8df-4ff2-bf59-7260d2705862 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.017238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 395/775] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:27 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:28.018233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f80749e-7402-4537-855a-a09d267b9851 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.026039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.026964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.027874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.028365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.035784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:28.039878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.041585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:28.057374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:28.058368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:28.059256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:28.059256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:28.059256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:28.061464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 381/776] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:28.069701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.204061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f80749e-7402-4537-855a-a09d267b9851 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.242377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.242991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.243272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.243545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.244175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:28.244479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.247704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.259259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f0198-8ad2-418d-a43d-e2b83b5e25eb None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.262157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 396/777] 10.222.0.26 () {56 vars in 1124 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:28.266424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.328359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.329418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.330033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.330498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.331332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:28.331829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.333693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.341250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:28.356314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 382/778] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:28.361346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.407651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f244cf-0a31-4c34-981f-2487a49d5c68 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.439022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:28.446970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.447637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.447923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.448196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.448827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:28.449167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.450523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.455923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 397/779] 10.222.0.26 () {56 vars in 1124 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:28.456712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.493337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae92ab-3dda-4c1b-826d-1702f35ea764 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.529081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.529717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.529993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.530260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.538237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.539772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:28.551205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:28.553201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:28.554137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 383/780] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/users/swift => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:28.555194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.587309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b0a80f-92cd-4fe8-bfd5-83393eb59105 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.590866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:28.633395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.634039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.634317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.634590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] PATH_INFO: `/v3/services/network` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.635510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.637264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:28.648485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:28.649520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:28.650363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:28.653007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 398/781] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/services/network => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:28.654017 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.698419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7414288-017d-4a9f-aff4-050d104bfe18 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.709564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:28.734477 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.735174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.735353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.735624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.736334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] RBAC: Adding query filter params (interface=public, service_id=7d833e3420fc4d4a90d7674ac2368063, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:28.736643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.738268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.751071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 384/782] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/endpoints?service_id=7d833e3420fc4d4a90d7674ac2368063&interface=public®ion_id=RegionOne => generated 194 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:28.751979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.785960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1881acf-ffe5-40ed-9d4d-0417578d152f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.828563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.829568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.830126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.830612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.831699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:28.832213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.834618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.855995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948a3485-2db1-4f23-a0d6-a7cebec49f54 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:28.856500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 399/783] 10.222.0.26 () {56 vars in 1135 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:28.857635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.897670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:28.903876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:28.905497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:28.906151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:28.906645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:28.907453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] RBAC: Adding query filter params (name=network) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:28.908438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:28.910918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:28.917215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 385/784] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/services?name=network => generated 124 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:28.918292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:28.967216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:28.997556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a81ff50-55a7-44c0-9f03-8cd8f5303ca3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.035428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970347b5-6c22-47c9-8c30-c810d9645a8a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.040010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.040629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.040910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.041213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.041818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.043198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.049669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 400/785] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/services => generated 772 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:29.057326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.079250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6569b793-ec61-4863-8c7e-6c26afd05e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.079997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6569b793-ec61-4863-8c7e-6c26afd05e3e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.080350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6569b793-ec61-4863-8c7e-6c26afd05e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.080692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6569b793-ec61-4863-8c7e-6c26afd05e3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.093113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6569b793-ec61-4863-8c7e-6c26afd05e3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.094582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6569b793-ec61-4863-8c7e-6c26afd05e3e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-6569b793-ec61-4863-8c7e-6c26afd05e3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:29.116345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:29.117546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:29.118418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:29.118418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:29.118418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:29.118846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 386/786] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:29.120008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9043bb-44e7-4ffb-a681-297c45b97816 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.124618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9043bb-44e7-4ffb-a681-297c45b97816 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.124618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9043bb-44e7-4ffb-a681-297c45b97816 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.124618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 387/787] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:29.124618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9043bb-44e7-4ffb-a681-297c45b97816 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.179531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:29.193709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e39baf5-0502-4bd8-b264-c297b8ccd296 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.244567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8381b8-7a98-4a75-8177-7bc1c5d9e287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.245471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8381b8-7a98-4a75-8177-7bc1c5d9e287 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.245928 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8381b8-7a98-4a75-8177-7bc1c5d9e287 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.246356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8381b8-7a98-4a75-8177-7bc1c5d9e287 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.247158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8381b8-7a98-4a75-8177-7bc1c5d9e287 None admin] RBAC: Adding query filter params (type=network) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:29.247309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:29.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8381b8-7a98-4a75-8177-7bc1c5d9e287 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.249817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8381b8-7a98-4a75-8177-7bc1c5d9e287 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.257006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 401/788] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/services?type=network => generated 350 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:29.258621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194368b1-a128-4103-b73c-2a9af658334d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.259110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194368b1-a128-4103-b73c-2a9af658334d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.259584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194368b1-a128-4103-b73c-2a9af658334d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.261159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9043bb-44e7-4ffb-a681-297c45b97816 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.280477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-194368b1-a128-4103-b73c-2a9af658334d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:29.281964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-194368b1-a128-4103-b73c-2a9af658334d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:29.305329 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.306198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.306652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.307073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.307875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:29.308183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.311223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.322847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 388/789] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:29.323697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.355954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 402/790] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:29.357317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59636fd-d18e-415b-b26c-63b57d44a9ff None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.357672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59636fd-d18e-415b-b26c-63b57d44a9ff None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.357949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59636fd-d18e-415b-b26c-63b57d44a9ff None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.363134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 403/791] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:29.363134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59636fd-d18e-415b-b26c-63b57d44a9ff None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.447778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c240cd09-b4b3-4d25-947d-c135cafb17fd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.460040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:29.484466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59636fd-d18e-415b-b26c-63b57d44a9ff None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.491968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd62f069-045a-4d98-98b6-4eaa7e341ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.492582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd62f069-045a-4d98-98b6-4eaa7e341ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.492859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd62f069-045a-4d98-98b6-4eaa7e341ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.493158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd62f069-045a-4d98-98b6-4eaa7e341ce0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.493867 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd62f069-045a-4d98-98b6-4eaa7e341ce0 None admin] RBAC: Adding query filter params (interface=public, service_id=9752e659fdea47bf97f36be7ea2c0232, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:29.494173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd62f069-045a-4d98-98b6-4eaa7e341ce0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.495732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd62f069-045a-4d98-98b6-4eaa7e341ce0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.504262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 389/792] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/endpoints?service_id=9752e659fdea47bf97f36be7ea2c0232&interface=public®ion_id=RegionOne => generated 194 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:29.504262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b6ce8c-8022-49d6-9fbe-d33404ca589c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.504262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b6ce8c-8022-49d6-9fbe-d33404ca589c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.504262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b6ce8c-8022-49d6-9fbe-d33404ca589c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.533555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-34b6ce8c-8022-49d6-9fbe-d33404ca589c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:29.534766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-34b6ce8c-8022-49d6-9fbe-d33404ca589c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:29.545458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:29.547457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.548784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.549466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.550246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.559558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.561023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:29.573481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:29.574396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:29.575289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 404/793] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:29.576226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.632951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 390/794] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:29.632951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b6ce8c-8022-49d6-9fbe-d33404ca589c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.714512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4069c8bb-5c8c-4b54-8ac1-80ff537cc7c2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.750180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.750952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.751339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.751685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.752416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.753908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.760180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 405/795] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/services => generated 772 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:29.761116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.763353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b6ce8c-8022-49d6-9fbe-d33404ca589c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.773360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:29.839937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.843228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.843514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.843810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.844536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:29.844843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:29.846378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:29.874381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 391/796] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:29.886015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:29.894088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:29.933704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07a436c-4eb4-4241-88ac-947d26fecf49 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:29.972695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:29.973273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:29.974436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:29.974715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:29.986992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:29.993029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:29.993979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:29.994827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 406/797] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:29.995534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.048627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c8a504-ad4c-4314-9f32-655870086890 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:30.077898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:30.101211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:30.101904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:30.102181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:30.102534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/3d69adfda7334567b03627a1d839f4f2/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:30.105226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=3d69adfda7334567b03627a1d839f4f2, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:30.106789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:30.123373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 392/798] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:29 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/3d69adfda7334567b03627a1d839f4f2/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:05:30.124307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.205389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:30.244403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6606c3c5-0ea6-4839-8835-7cbcd5eba773 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:30.284947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:30.285566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:30.285844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:30.286184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:30.287268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:30.287768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:30.289245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:30.300336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 393/799] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:30.306693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.325109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3217767-17af-43c5-bea0-83db46408d42 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:30.393753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:30.394496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:30.394779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:30.395048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:30.407405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:30.413810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:30.414763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:30.415700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 407/800] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 420 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:30.426356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.431285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5333f36d-0868-4bee-928b-4bc04f07c60c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:30.472551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:30.474325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:30.474768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:30.475195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:30.485136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:30.490275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:30.491750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:30.526014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:30.527159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:30.528285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:30.528285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:30.528285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:30.531460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 394/801] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:30.537994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.572356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e911fd8-1e11-4de2-8e28-2d52a6228e44 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:30.605466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:30.616540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:30.617512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:30.617938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:30.618492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:30.619309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:30.619984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:30.621525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:30.633856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 408/802] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:30.640096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.758900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b234df16-5e84-451a-b996-b95febd6f312 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:30.762918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c5512-a3b1-4f04-8c8c-b15f6de3cdc5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:30.805548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:30.806525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:30.807711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:30.808211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:30.822919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:30.823569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:30.823848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:30.824187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:30.824769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:30.825130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:30.825576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:30.826499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:30.827429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:30.839529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 395/803] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:30.844907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:30.847107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:30.848015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:30.850239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:30.850239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:30.850239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:30.851030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 409/804] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:30.857421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:30.933732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:30.938455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:31.006210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120b1ff6-091e-4181-8842-80b07365cc70 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.042989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.044064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.044345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.044622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.045362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:31.045668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.047030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.060464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 410/805] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:31.063105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52dfe7-e188-467b-9326-39e213bee7b0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.066199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:31.127608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.128581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.129337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.129868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.140499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.142948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:31.158014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:31.159263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:05:31.160198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:31.163057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 396/806] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:05:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:31.169907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:31.300671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b626e7f-18ac-4ec8-a469-f961f7ccdaa0 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.304740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:31.311704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:31.330932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fcb1a3-e8c7-4272-a2ca-419590f721ba None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.346186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.346842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.347121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.347392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.356244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.357729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:31.370523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:31.371496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:31.372425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 411/807] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:05:31 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:31.377337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:31.477650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.478272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.478550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.478819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.479550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:31.479853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.481292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.484041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cecc42-ad83-47fe-91e5-bae000db7998 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.520160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.520160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.520734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.520916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.521924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:31.522157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.523650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.538916 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 397/808] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:05:31 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:31.545906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:31.547078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 412/809] 10.222.0.26 () {56 vars in 1149 bytes} [Sat Oct 1 04:05:31 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:31.552987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:31.585431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:31.669835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a517b1-93af-4382-84ed-15ada40aa5b5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.707235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.708623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.708906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.709211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.723698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.725223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:31.742193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:31.743189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:31.744086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:31.744086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:31.744086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:31.744086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 413/810] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:31.749318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:31.785074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e1f517-a050-4277-b845-b2d0b8808967 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.822936 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5beed79-2602-4a37-b6df-beacdac2de9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.823929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5beed79-2602-4a37-b6df-beacdac2de9c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.824416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5beed79-2602-4a37-b6df-beacdac2de9c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.824909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5beed79-2602-4a37-b6df-beacdac2de9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.831085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:31.855949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:31.869396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f30a40-512b-48e6-acfd-76592c1e613c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:31.908852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5beed79-2602-4a37-b6df-beacdac2de9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.910699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5beed79-2602-4a37-b6df-beacdac2de9c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.911194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5ae5b-792d-49b5-be38-7913101ff718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:31.911792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5ae5b-792d-49b5-be38-7913101ff718 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.912072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5ae5b-792d-49b5-be38-7913101ff718 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.912445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5ae5b-792d-49b5-be38-7913101ff718 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.913215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5ae5b-792d-49b5-be38-7913101ff718 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:31.913524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5ae5b-792d-49b5-be38-7913101ff718 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:31.916034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5ae5b-792d-49b5-be38-7913101ff718 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:31.936063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 414/811] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:31.938299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16fcd58-fbec-467e-ac62-3d0be0fdd371 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:31.938654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16fcd58-fbec-467e-ac62-3d0be0fdd371 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:31.939016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16fcd58-fbec-467e-ac62-3d0be0fdd371 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:31.945493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 415/812] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:31.945493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16fcd58-fbec-467e-ac62-3d0be0fdd371 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.031795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 398/813] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:31 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:05:32.033687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d087ba7-8f12-432e-9e58-e12ff18c2caf None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.034152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d087ba7-8f12-432e-9e58-e12ff18c2caf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.034591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d087ba7-8f12-432e-9e58-e12ff18c2caf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.054679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6d087ba7-8f12-432e-9e58-e12ff18c2caf None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:32.056262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6d087ba7-8f12-432e-9e58-e12ff18c2caf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:32.069006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16fcd58-fbec-467e-ac62-3d0be0fdd371 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.104104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:32.110250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.111330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.111714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.112083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.117796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:32.121159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.122694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:32.137711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:32.138703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:32.139650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.144234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 416/814] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:32.145427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.151891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 399/815] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:32.153286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d087ba7-8f12-432e-9e58-e12ff18c2caf None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.266027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88856ab-60ed-4636-ad1f-adecb8fbbfe0 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.307905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d087ba7-8f12-432e-9e58-e12ff18c2caf None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.308129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c2da9-a61f-4a6f-aaca-9c069569c869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.308640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2da9-a61f-4a6f-aaca-9c069569c869 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.308885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2da9-a61f-4a6f-aaca-9c069569c869 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.309199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2da9-a61f-4a6f-aaca-9c069569c869 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.320887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c2da9-a61f-4a6f-aaca-9c069569c869 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.322453 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c2da9-a61f-4a6f-aaca-9c069569c869 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-f73c2da9-a61f-4a6f-aaca-9c069569c869 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:32.338075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:32.339070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:32.339965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:32.339965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:32.339965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.339965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 417/816] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:32.343873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a7d33-a30b-4c60-a4e4-9d8d7b5a64c7 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.343873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a7d33-a30b-4c60-a4e4-9d8d7b5a64c7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.343873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a7d33-a30b-4c60-a4e4-9d8d7b5a64c7 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.347099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 418/817] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:32.347984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a7d33-a30b-4c60-a4e4-9d8d7b5a64c7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.353632 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131d8775-6670-4359-9399-f6b9ab24e03f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.354261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131d8775-6670-4359-9399-f6b9ab24e03f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.354541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131d8775-6670-4359-9399-f6b9ab24e03f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.354817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131d8775-6670-4359-9399-f6b9ab24e03f None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.355514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131d8775-6670-4359-9399-f6b9ab24e03f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:32.355820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131d8775-6670-4359-9399-f6b9ab24e03f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.357277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131d8775-6670-4359-9399-f6b9ab24e03f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.373485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 400/818] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:32.375735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e5d5c5-c73e-4554-9dc1-8a6c09e309a4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.376019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e5d5c5-c73e-4554-9dc1-8a6c09e309a4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.376286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e5d5c5-c73e-4554-9dc1-8a6c09e309a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.398456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-76e5d5c5-c73e-4554-9dc1-8a6c09e309a4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:32.399635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-76e5d5c5-c73e-4554-9dc1-8a6c09e309a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:32.406440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:32.444242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:32.475926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 401/819] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:32.477039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e5d5c5-c73e-4554-9dc1-8a6c09e309a4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.482198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a7d33-a30b-4c60-a4e4-9d8d7b5a64c7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.523771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.524663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.524959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.525252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.526086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:32.526391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.528390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.544199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 419/820] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:32.545796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.610972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e5d5c5-c73e-4554-9dc1-8a6c09e309a4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.647490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.648161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.648451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.648726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/98e0eece1e7146c48cf68b599e58c0bf/roles/8f8d9a3eff6c47faa94d854a3f42be35` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.664151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=98e0eece1e7146c48cf68b599e58c0bf, role_id=8f8d9a3eff6c47faa94d854a3f42be35)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.665659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.682258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455e193d-c4ee-4b23-a0a6-e17e813eb0df None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.685006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 402/821] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:32 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/98e0eece1e7146c48cf68b599e58c0bf/roles/8f8d9a3eff6c47faa94d854a3f42be35 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:05:32.685920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.691377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:32.731418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:32.742779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5814256-903e-4374-98c0-7394a6a3b20c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.743693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5814256-903e-4374-98c0-7394a6a3b20c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.743985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5814256-903e-4374-98c0-7394a6a3b20c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.745010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5814256-903e-4374-98c0-7394a6a3b20c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.745666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5814256-903e-4374-98c0-7394a6a3b20c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.747084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5814256-903e-4374-98c0-7394a6a3b20c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-e5814256-903e-4374-98c0-7394a6a3b20c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:32.765902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:32.766979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.767849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 420/822] 10.222.0.26 () {56 vars in 1129 bytes} [Sat Oct 1 04:05:32 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:32.769227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc115da-eaa8-481e-b353-c6e8cc2d6481 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.769227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc115da-eaa8-481e-b353-c6e8cc2d6481 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.769394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc115da-eaa8-481e-b353-c6e8cc2d6481 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.772575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 421/823] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:32.773702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc115da-eaa8-481e-b353-c6e8cc2d6481 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.828325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c759ad-6ded-4270-8429-b63495872310 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.866141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.866794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.867079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.867346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.876657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.878429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:32.894809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:32.895815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 403/824] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:32.896715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:32.902526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc115da-eaa8-481e-b353-c6e8cc2d6481 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:32.941474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b90ea76-36d6-4716-b5f4-bb921807c280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:32.942631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b90ea76-36d6-4716-b5f4-bb921807c280 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.942918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b90ea76-36d6-4716-b5f4-bb921807c280 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.943181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b90ea76-36d6-4716-b5f4-bb921807c280 None admin] PATH_INFO: `/v3/services/network` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.943765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b90ea76-36d6-4716-b5f4-bb921807c280 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:32.945136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b90ea76-36d6-4716-b5f4-bb921807c280 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-3b90ea76-36d6-4716-b5f4-bb921807c280 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:32.954251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:32.955258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:32.956096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 422/825] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:32 2022] GET /identity/v3/services/network => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:32.956957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2a6a3c-7c20-44c7-8e6c-ab5f248a086a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:32.956957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2a6a3c-7c20-44c7-8e6c-ab5f248a086a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:32.957100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2a6a3c-7c20-44c7-8e6c-ab5f248a086a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:32.975313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ae2a6a3c-7c20-44c7-8e6c-ab5f248a086a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:32.976497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ae2a6a3c-7c20-44c7-8e6c-ab5f248a086a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:32.993855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.005442 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.013488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d42e4a7-8d20-44b3-aef3-e65fe56fbe4c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.047799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.048544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.048830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.049141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.049777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:33.050086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.051504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.056956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 423/826] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:33.056956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2a6a3c-7c20-44c7-8e6c-ab5f248a086a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.068678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 404/827] 10.222.0.26 () {56 vars in 1124 bytes} [Sat Oct 1 04:05:32 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:33.069655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.184484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d96be-a7bb-4e3a-ac09-f4c2f6274346 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.199072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2a6a3c-7c20-44c7-8e6c-ab5f248a086a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.228744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.229391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.229674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.229949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.230633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] RBAC: Adding query filter params (name=network) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:33.230989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.232351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.237420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 405/828] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:33 2022] GET /identity/v3/services?name=network => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:33.238272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.245756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.246372 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.246653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.246930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.247629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:33.247937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.249403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.261796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.263485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 424/829] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:33.264350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.296330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.355077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311900b3-c79d-47f0-8c6d-bc300da25136 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.393248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72af4d12-e477-459e-8bd1-a6514eada58e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.394492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.395243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.395792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.396193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.396840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:33.397179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.398545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.407967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 406/830] 10.222.0.26 () {56 vars in 1124 bytes} [Sat Oct 1 04:05:33 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:33.407967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.434853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.435442 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.435740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.435995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.436638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] RBAC: Adding query filter params (type=network) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:33.437013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.438366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.447649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 425/831] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:33 2022] GET /identity/v3/services?type=network => generated 350 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:33.448710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.493443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.524403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2f9a05-0c28-460a-857a-7fcc74b86fe6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.546273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.561932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.562547 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.562832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.563109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.563838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=901b9bb47c2f40c2b1b4cfe7886a1167) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:33.564150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.565321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1e753f-bb1e-4ea8-8874-843f0be5be08 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.565634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.571827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 407/832] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:33 2022] GET /identity/v3/role_assignments?user.id=901b9bb47c2f40c2b1b4cfe7886a1167&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:33.572617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.619033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.619678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.619961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.620513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.633305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:33.638274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:33.639309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:33.640171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 426/833] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:33.640865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.710999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f3002e-f56e-4a16-988e-89ad43081092 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.747966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.748630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.748915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.749213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.749891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.751259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.770782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.784119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 408/834] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:05:33 2022] POST /identity/v3/endpoints => generated 346 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:33.785596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:33.794800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403597b-6fc6-4529-b841-d6ea60bf9c60 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.852223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.853605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.854222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.854642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.855616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.858103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:33.884845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:33.894670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 427/835] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:05:33 2022] POST /identity/v3/endpoints => generated 337 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:33.936226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf1045-a8f2-4eea-beb7-bef89a4458aa None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:33.985779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:33.986670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:33.987125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:33.987558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:33.988410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:33.989432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:33.991013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:34.003156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 409/836] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:34.009721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:34.124845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43b530b-4b16-45e3-b661-c7794a260b76 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:34.162072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:34.162721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:34.163005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:34.163275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:34.174702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:34.176314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:34.194493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:34.195518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:34.196413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:34.196413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:34.196413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:34.196413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 428/837] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:34.202090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:34.268716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:34.322146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861a7313-c8ab-458a-9557-a8c8841f703f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:34.358624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:34.359596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:34.360083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:34.360618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:34.361487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:34.362027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:34.363656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:34.377410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 410/838] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:34.382313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:34.450531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:34.512996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4553d7b4-0315-483a-8034-6885107dac88 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:34.548997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:34.549935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:34.550392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:34.550866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:34.559682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:34.561394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:34.574076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:34.575083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 01 04:05:34.576025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:34.578336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 429/839] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:05:34 2022] GET /identity/v3/users/swift => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:34.584454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:34.696604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc0cbf-4497-4a4b-b09d-c2af46f336c3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:34.714095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:34.734761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20db4055-2163-4264-9340-6dc4c3e667ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:34.735379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db4055-2163-4264-9340-6dc4c3e667ad None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:34.735660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db4055-2163-4264-9340-6dc4c3e667ad None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:34.735939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20db4055-2163-4264-9340-6dc4c3e667ad None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:34.736669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20db4055-2163-4264-9340-6dc4c3e667ad None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:34.737003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20db4055-2163-4264-9340-6dc4c3e667ad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:34.738430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20db4055-2163-4264-9340-6dc4c3e667ad None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:34.757288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 411/840] 10.222.0.26 () {56 vars in 1135 bytes} [Sat Oct 1 04:05:34 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:34.762819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:34.876959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7ae541-e320-42fe-8e41-00d6a700e5b1 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:34.888266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:34.913721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:34.914327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:34.914606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:34.914879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:34.916348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd883db-04dd-464f-b661-9681cf680056 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:34.916636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd883db-04dd-464f-b661-9681cf680056 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:34.916908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd883db-04dd-464f-b661-9681cf680056 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:34.919652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 412/841] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:34.923597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec555a1a-50be-4c2c-b296-153fc4a22e32 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:34.923889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec555a1a-50be-4c2c-b296-153fc4a22e32 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:34.924165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec555a1a-50be-4c2c-b296-153fc4a22e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:34.943981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ec555a1a-50be-4c2c-b296-153fc4a22e32 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:34.945204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ec555a1a-50be-4c2c-b296-153fc4a22e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:34.996747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:34.998324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:35.029030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 413/842] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:34 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:35.116602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 430/843] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:34 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:05:35.122005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec555a1a-50be-4c2c-b296-153fc4a22e32 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.221974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec555a1a-50be-4c2c-b296-153fc4a22e32 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.225657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:35.259281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:35.259925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:35.260206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:35.260483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:35.270881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:35.272467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:35.291588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:35.292548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:35.293610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:35.293610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:35.293610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:35.293610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 414/844] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:35.301585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.314037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:35.322032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.407602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb258ede-6cf1-4674-b023-adb07f881e02 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.431149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd9b36-9c2e-4fff-a996-a7c22156aeee None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.445291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:35.446278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:35.446811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:35.447278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:35.448222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:35.448818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:35.450363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:35.463636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 431/845] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:35.470163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.476787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:35.477721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:35.478195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:35.478679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:35.493536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:35.499370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:35.500288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:05:35.501493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:35.503979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 415/846] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:35.510056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.534475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:35.556315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:35.597177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f5b62-0d35-400d-9944-e054bcb263f2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.626954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2db9629-a811-45ea-a932-2bf3a7510589 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.640581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:35.641242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:35.641579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:35.641864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:35.651541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:35.653648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:35.663266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:35.664139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:35.664618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:35.665126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:35.665990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:35.666487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:35.667960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:35.668823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:35.669767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:35.670882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:35.673065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 432/847] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:35.674797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.689330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 416/848] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:35.691248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.774835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:35.779387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c47ea2-21a5-4fe5-8bc0-95cfbbf5c073 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.799619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996473ed-94ca-4028-b03b-1dff39db0688 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.810627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:35.818898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:35.819806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:35.820281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:35.820747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:35.821646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:35.822215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:35.823809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:35.837303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:35.837668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 433/849] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:35.837936 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:35.838216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:35.838487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:35.844407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.856522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:35.858105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:35.874067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:35.875119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:35.876016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:35.876016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:35.876016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:35.880220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 417/850] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:35.881315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:35.982799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3765c6db-c00d-47e8-8b30-6f996894b087 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:35.993865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b2299f-1c8c-4c77-a911-acf3f7fd5e1a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:36.022655 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.023266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.023549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.023821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.024341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:36.025535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=3d69adfda7334567b03627a1d839f4f2) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:36.026112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.027780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.031789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce5201b-7f00-4f06-8008-57d02d1e6e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.032644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce5201b-7f00-4f06-8008-57d02d1e6e70 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.033348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce5201b-7f00-4f06-8008-57d02d1e6e70 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.033761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce5201b-7f00-4f06-8008-57d02d1e6e70 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.034540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce5201b-7f00-4f06-8008-57d02d1e6e70 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:36.035044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 434/851] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/role_assignments?user.id=3d69adfda7334567b03627a1d839f4f2&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:36.035219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce5201b-7f00-4f06-8008-57d02d1e6e70 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.040997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce5201b-7f00-4f06-8008-57d02d1e6e70 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.056852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 418/852] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:36.058384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:36.130600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:36.182379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bd7270-762b-4de5-bd99-3b1d51f6040e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:36.214275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323a830-e0b4-498b-9b64-49fa7e0e6521 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.214879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323a830-e0b4-498b-9b64-49fa7e0e6521 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.215349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323a830-e0b4-498b-9b64-49fa7e0e6521 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.216862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 419/853] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:36.220483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.221600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.221600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.221766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.223554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a2b792-51c0-499e-9a1e-3c7192e550ae None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.224029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a2b792-51c0-499e-9a1e-3c7192e550ae None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.224123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a2b792-51c0-499e-9a1e-3c7192e550ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.233340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.234983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.245899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e9a2b792-51c0-499e-9a1e-3c7192e550ae None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:36.247085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e9a2b792-51c0-499e-9a1e-3c7192e550ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:36.247196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:36.248081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:36.249083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 435/854] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:36.255035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:36.351863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 420/855] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:36 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:36.391418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53b3161-bf38-44b0-bcbe-03bbcd41efc7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:36.399543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d32d54e-41cf-45fa-8ef6-1d19ca3e0f11 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.399825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d32d54e-41cf-45fa-8ef6-1d19ca3e0f11 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.400105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d32d54e-41cf-45fa-8ef6-1d19ca3e0f11 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.403176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 421/856] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:36.407661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:36.409121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c4f28b-8bcb-4583-8f1f-1e6532d58429 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.409409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c4f28b-8bcb-4583-8f1f-1e6532d58429 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.409685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c4f28b-8bcb-4583-8f1f-1e6532d58429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.430650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f7c4f28b-8bcb-4583-8f1f-1e6532d58429 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:36.432270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f7c4f28b-8bcb-4583-8f1f-1e6532d58429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:36.436327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.436957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.437247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.437519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.438314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:36.438681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.440159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.463970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 436/857] 10.222.0.26 () {56 vars in 1149 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:36.465959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:36.520971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 422/858] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:36 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:36.587398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827b4588-4bc5-4629-b635-3cf2be4a13b6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:36.624726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.625285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.625567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.625843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.639364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c4f28b-8bcb-4583-8f1f-1e6532d58429 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:36.685319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:36.702060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:36.705412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.708858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.771448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c4f28b-8bcb-4583-8f1f-1e6532d58429 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:36.806494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.807129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.807405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.807673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.808262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.809642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:36.818920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:36.820067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:36.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 423/859] 10.222.0.26 () {56 vars in 1132 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:36.822042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:36.830751 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 437/860] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 01 04:05:36.831457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:36.908995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:36.934899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85665cb2-fc5d-4b0b-bd51-4b89fe8ed677 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:36.943217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7674b091-a786-4983-b789-70c84c43f624 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:36.949947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:36.972862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.973615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.973903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.974177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.974772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.976134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.982120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:36.982729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:36.983008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:36.983280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:36.983902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:36.984205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:36.985027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:36.986049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:36.986913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:36.987990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:36.991540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 424/861] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:36.991980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 438/862] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:36.992393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:36.992884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:37.102485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a48724-91a3-482c-9366-bd30cef8622f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:37.110535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1770ab-cab4-426d-8d70-b568ec9fc267 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:37.136749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:37.137415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:37.137689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:37.138225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:37.146681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:37.147279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:37.147761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:37.147870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:37.148463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:37.148767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:37.149704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:37.150152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:37.151388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:37.152392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:37.161739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 439/863] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:37.162585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:37.170957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:37.172010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:37.172890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:37.172890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:37.172890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:37.172890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 425/864] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:37.173852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:37.205934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:37.268254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a629d829-df74-465c-96c2-a49678cf9146 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:37.283499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5d060a-5b52-4860-9e17-91ebc89bb227 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:37.313621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:37.314419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:37.314810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:37.315075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:37.315689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:37.315984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:37.317603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:37.332959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 440/865] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:37 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:37.332959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:37.334489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:37.335210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:37.335568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:37.335918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:37.336648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:37.337074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:37.338539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:37.350391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 426/866] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:37 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:37.413429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:37.459776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab00734-3a1b-4540-948b-0dfe109f503a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:37.506083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:37.507476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:37.507902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:37.508332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:37.509164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:37.509612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:37.511194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:37.525884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 441/867] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:37.532516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:37.638316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:37.646149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86181735-0ed6-4237-a633-f63c0b7e25e0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:37.682315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:37.682957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:37.683239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:37.683573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:37.690605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:37.693055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:37.704710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:37.705820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:37.706911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 427/868] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:37.711712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:37.814053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2528d971-a79e-4602-b2db-3fd52cc8c426 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:37.842292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:37.854409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fe7508-555f-4806-a188-f2fa10eee219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:37.855015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fe7508-555f-4806-a188-f2fa10eee219 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:37.855291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fe7508-555f-4806-a188-f2fa10eee219 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:37.855559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fe7508-555f-4806-a188-f2fa10eee219 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:37.856251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fe7508-555f-4806-a188-f2fa10eee219 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:37.856554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fe7508-555f-4806-a188-f2fa10eee219 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:37.857988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fe7508-555f-4806-a188-f2fa10eee219 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:37.872461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 442/869] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:37.878184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:37.991256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f8c327-1631-4d06-b6a2-812c006dc124 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:38.025939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43daf50-aae1-4f41-9785-794c3b1bf8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:38.026598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43daf50-aae1-4f41-9785-794c3b1bf8ce None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:38.026880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43daf50-aae1-4f41-9785-794c3b1bf8ce None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:38.027152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43daf50-aae1-4f41-9785-794c3b1bf8ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:38.027920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43daf50-aae1-4f41-9785-794c3b1bf8ce None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=98e0eece1e7146c48cf68b599e58c0bf) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:38.028230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43daf50-aae1-4f41-9785-794c3b1bf8ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:38.029666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43daf50-aae1-4f41-9785-794c3b1bf8ce None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:38.036665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 428/870] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:37 2022] GET /identity/v3/role_assignments?user.id=98e0eece1e7146c48cf68b599e58c0bf&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:38.421129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b7bfdd-b01d-4a55-b663-b3787ded1c2a None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:38.421129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b7bfdd-b01d-4a55-b663-b3787ded1c2a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:38.421494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b7bfdd-b01d-4a55-b663-b3787ded1c2a None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:38.423040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 443/871] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:38.429283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1863d0-4892-4b94-af2c-d7e8fe0a4ec5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:38.429735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1863d0-4892-4b94-af2c-d7e8fe0a4ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:38.430015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1863d0-4892-4b94-af2c-d7e8fe0a4ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:38.449273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9c1863d0-4892-4b94-af2c-d7e8fe0a4ec5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:38.450461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9c1863d0-4892-4b94-af2c-d7e8fe0a4ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:38.464012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:38.520767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 429/872] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:38 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:38.790263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b7bfdd-b01d-4a55-b663-b3787ded1c2a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:38.900545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b7bfdd-b01d-4a55-b663-b3787ded1c2a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:38.937564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:38.938302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:38.938586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:38.938862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:38.939449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:38.940853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:38.949948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:38.952269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:38.954248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 444/873] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:38.960976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1863d0-4892-4b94-af2c-d7e8fe0a4ec5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:39.082805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1863d0-4892-4b94-af2c-d7e8fe0a4ec5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:39.121100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd52a03-b922-4942-9e87-e2aac14e60ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:39.122092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd52a03-b922-4942-9e87-e2aac14e60ab None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:39.122568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd52a03-b922-4942-9e87-e2aac14e60ab None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:39.123036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd52a03-b922-4942-9e87-e2aac14e60ab None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:39.123879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd52a03-b922-4942-9e87-e2aac14e60ab None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:39.124375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd52a03-b922-4942-9e87-e2aac14e60ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:39.125961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd52a03-b922-4942-9e87-e2aac14e60ab None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:39.132285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 430/874] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:39.141013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:39.193829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:39.255410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a88c8-b122-4691-86a7-3df79de030ce None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:39.300415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f82d6b-b08c-4526-bd1a-b9b703c7fb31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:39.301068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f82d6b-b08c-4526-bd1a-b9b703c7fb31 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:39.301935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f82d6b-b08c-4526-bd1a-b9b703c7fb31 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:39.302150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f82d6b-b08c-4526-bd1a-b9b703c7fb31 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:39.304096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f82d6b-b08c-4526-bd1a-b9b703c7fb31 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:39.304537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f82d6b-b08c-4526-bd1a-b9b703c7fb31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:39.306032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f82d6b-b08c-4526-bd1a-b9b703c7fb31 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:39.311726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 445/875] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:39 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:39.710581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa526a63-7079-47af-af7b-6f7dbb357933 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:39.711596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa526a63-7079-47af-af7b-6f7dbb357933 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:39.712109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa526a63-7079-47af-af7b-6f7dbb357933 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:39.713991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 431/876] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:39.720178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4927f-a589-41c1-a6e0-5008dc7cc816 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:39.720776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4927f-a589-41c1-a6e0-5008dc7cc816 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:39.721270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e4927f-a589-41c1-a6e0-5008dc7cc816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:39.744035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-81e4927f-a589-41c1-a6e0-5008dc7cc816 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:39.745764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-81e4927f-a589-41c1-a6e0-5008dc7cc816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:39.796207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bcca1-cd2b-4c24-aa98-79750802b4cb None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:39.796408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bcca1-cd2b-4c24-aa98-79750802b4cb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:39.796670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bcca1-cd2b-4c24-aa98-79750802b4cb None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:39.798410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 432/877] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:39.804976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd709da8-a948-46d8-b1ae-7dc4678ce995 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:39.805543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd709da8-a948-46d8-b1ae-7dc4678ce995 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:39.806033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd709da8-a948-46d8-b1ae-7dc4678ce995 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:39.827402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 446/878] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:39 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:39.829299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-dd709da8-a948-46d8-b1ae-7dc4678ce995 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:39.830892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dd709da8-a948-46d8-b1ae-7dc4678ce995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:39.842418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:39.917652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 433/879] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:39 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:40.103553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e4927f-a589-41c1-a6e0-5008dc7cc816 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:40.146749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:40.209088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e4927f-a589-41c1-a6e0-5008dc7cc816 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:40.221066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd709da8-a948-46d8-b1ae-7dc4678ce995 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:40.246045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8141e740-8f57-4387-b215-ae83b9c92197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:40.246771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8141e740-8f57-4387-b215-ae83b9c92197 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:40.247046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8141e740-8f57-4387-b215-ae83b9c92197 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:40.247315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8141e740-8f57-4387-b215-ae83b9c92197 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:40.247969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8141e740-8f57-4387-b215-ae83b9c92197 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:40.249529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8141e740-8f57-4387-b215-ae83b9c92197 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:40.267329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 447/880] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:40 2022] POST /identity/v3/services => generated 269 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:40.344309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd709da8-a948-46d8-b1ae-7dc4678ce995 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:40.394137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf41d1c-2fb8-42d4-a70a-6a27372fdc1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:40.395721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf41d1c-2fb8-42d4-a70a-6a27372fdc1e None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:40.396358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf41d1c-2fb8-42d4-a70a-6a27372fdc1e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:40.396856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf41d1c-2fb8-42d4-a70a-6a27372fdc1e None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:40.397815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf41d1c-2fb8-42d4-a70a-6a27372fdc1e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:40.399457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf41d1c-2fb8-42d4-a70a-6a27372fdc1e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:40.413563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:40.421353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 434/881] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:40 2022] POST /identity/v3/services => generated 251 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:05:40.512210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b661145-f120-491d-abf2-35f50c88d445 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:40.512566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b661145-f120-491d-abf2-35f50c88d445 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:40.512778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b661145-f120-491d-abf2-35f50c88d445 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:40.514461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 448/882] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:40.521245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2d84a4-e90d-4194-9f95-5cfb7830b7cb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:40.521992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2d84a4-e90d-4194-9f95-5cfb7830b7cb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:40.521992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2d84a4-e90d-4194-9f95-5cfb7830b7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:40.544303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-de2d84a4-e90d-4194-9f95-5cfb7830b7cb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:40.545557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-de2d84a4-e90d-4194-9f95-5cfb7830b7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:40.632061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 435/883] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:40 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:40.908442 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b661145-f120-491d-abf2-35f50c88d445 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:41.020850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b661145-f120-491d-abf2-35f50c88d445 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:41.022437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:41.057624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:41.058250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.058527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.058797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.069134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:41.073907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:41.074940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:41.075883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 449/884] 10.222.0.26 () {56 vars in 1120 bytes} [Sat Oct 1 04:05:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:41.082231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2d84a4-e90d-4194-9f95-5cfb7830b7cb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:41.192649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2d84a4-e90d-4194-9f95-5cfb7830b7cb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:41.217449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:41.229313 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:41.229955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.230219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.230492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.231194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:41.231467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:41.232844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:41.242851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 436/885] 10.222.0.26 () {56 vars in 1115 bytes} [Sat Oct 1 04:05:41 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:41.248624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:41.351343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954947fc-b4ac-4ed4-9535-c6b326caf8d9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:41.385977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:41.386949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.387436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.387922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.399031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:41.400730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:41.415516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:41.416570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:41.417619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:41.417619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:41.417619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:41.420054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 450/886] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:41.426695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:41.542931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9ceb20-f072-41c6-92c7-93489ba2f5db None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:41.576996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacc20d9-82fc-4fd7-9c32-4087579d8c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:41.577947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacc20d9-82fc-4fd7-9c32-4087579d8c08 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.578433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacc20d9-82fc-4fd7-9c32-4087579d8c08 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.578908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacc20d9-82fc-4fd7-9c32-4087579d8c08 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.579748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacc20d9-82fc-4fd7-9c32-4087579d8c08 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:41.580256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacc20d9-82fc-4fd7-9c32-4087579d8c08 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:41.581839 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacc20d9-82fc-4fd7-9c32-4087579d8c08 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:41.595253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 437/887] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:41.600802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:41.619602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:41.713998 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed08003-fee2-4a01-b080-02c6d8a223b8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:41.723999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6b8c4-f4ae-4d18-84f9-d081d6ea08d9 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.724298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6b8c4-f4ae-4d18-84f9-d081d6ea08d9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.724553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6b8c4-f4ae-4d18-84f9-d081d6ea08d9 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.726631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 438/888] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:41.731723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f706b10f-62a3-4188-a168-c40bc88b4277 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.732011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f706b10f-62a3-4188-a168-c40bc88b4277 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.732292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f706b10f-62a3-4188-a168-c40bc88b4277 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.753881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:41.754577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f706b10f-62a3-4188-a168-c40bc88b4277 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:41.755126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.755604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.755737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f706b10f-62a3-4188-a168-c40bc88b4277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:41.756202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.764078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:41.765790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:41.777314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:41.778270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:41.779242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:41.781752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 451/889] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:05:41 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:41.790541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:41.817641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:41.837013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 439/890] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:41 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:41.881811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:41.906809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ff9c1-db36-440f-a2cf-f7744487519f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:41.941421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:41.942809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:41.943301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:41.943799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:41.944765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:41.945287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:41.946902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:41.964991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 452/891] 10.222.0.26 () {56 vars in 1149 bytes} [Sat Oct 1 04:05:41 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:41.974979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f706b10f-62a3-4188-a168-c40bc88b4277 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:42.082232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f706b10f-62a3-4188-a168-c40bc88b4277 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:42.117493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:42.118140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.118381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.118637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.131368 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:42.169283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:42.191357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:42.193723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:42.233673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da2a82-29e3-46be-b1d6-5017a70aeb0b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:42.264431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:42.269439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:42.270672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.271189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.271637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.272467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:42.273975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:42.297108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 453/892] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:42 2022] POST /identity/v3/services => generated 241 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:42.304014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 440/893] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:41 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:05:42.309857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:42.436543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61853ff-d623-42c9-b942-686ee18e0e31 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:42.475506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81af03b4-9113-4457-a511-44b418ebb54b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:42.476155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81af03b4-9113-4457-a511-44b418ebb54b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.476468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81af03b4-9113-4457-a511-44b418ebb54b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.476739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81af03b4-9113-4457-a511-44b418ebb54b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.490401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81af03b4-9113-4457-a511-44b418ebb54b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:42.490921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81af03b4-9113-4457-a511-44b418ebb54b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-81af03b4-9113-4457-a511-44b418ebb54b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:42.507512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:42.508538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:42.509490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:42.509490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:42.509490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:42.509490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 454/894] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:42.516336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:42.601972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:42.619576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9a41d9-6688-46bc-acd7-ac6905ae7646 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.619882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9a41d9-6688-46bc-acd7-ac6905ae7646 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.620132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9a41d9-6688-46bc-acd7-ac6905ae7646 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.621769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 455/895] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:42.627032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d8ebfa-c5d8-49fe-bd5a-45ab4f711d09 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.627314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d8ebfa-c5d8-49fe-bd5a-45ab4f711d09 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.627590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d8ebfa-c5d8-49fe-bd5a-45ab4f711d09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.628362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f9f502-b46e-4750-9174-55043f310390 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:42.646298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-46d8ebfa-c5d8-49fe-bd5a-45ab4f711d09 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:42.647535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-46d8ebfa-c5d8-49fe-bd5a-45ab4f711d09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:42.670362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:42.670974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.671703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.671978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.672974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:42.673280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:42.675156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:42.689455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:42.689920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 441/896] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:42.698474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:42.733260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 456/897] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:42.812401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c14518-dd0e-4378-8514-eae23c213526 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:42.844057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee7dcf-c77a-4d96-bb43-8eb437864bd3 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.844413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee7dcf-c77a-4d96-bb43-8eb437864bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.844611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee7dcf-c77a-4d96-bb43-8eb437864bd3 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.846234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 457/898] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:42.848799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:42.849647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.849930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.850210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.851759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1e24b-3977-4820-a4c3-08c5ede7094a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:42.852054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1e24b-3977-4820-a4c3-08c5ede7094a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:42.852331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1e24b-3977-4820-a4c3-08c5ede7094a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:42.853166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:42.858392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:42.859884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:42.871421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-24c1e24b-3977-4820-a4c3-08c5ede7094a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:42.871950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:42.872911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:42.873844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 442/899] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:42.875791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-24c1e24b-3977-4820-a4c3-08c5ede7094a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:42.880626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:42.965038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 458/900] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:42.998895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9af44e8-a66d-44ab-8be5-a653234b65d3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.010991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c1e24b-3977-4820-a4c3-08c5ede7094a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.037460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.038122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.038409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.038684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.039378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.039689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.041323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.055704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 443/901] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:43.061475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.121428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:43.124788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:43.126243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c1e24b-3977-4820-a4c3-08c5ede7094a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.165317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.165955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.166233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.166506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.167086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.168544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.171968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6844af5-d6e3-40d7-a6a2-0ed0314bda26 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:43.178604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:43.179839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:43.180811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 459/902] 10.222.0.26 () {56 vars in 1132 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:43.185747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.214689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.215921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.216548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.217028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.218559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] RBAC: Adding query filter params (role.id=16de5c1385014299a8e7cef6b0114795, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=98e0eece1e7146c48cf68b599e58c0bf) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.219052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.221453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.239406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 444/903] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/role_assignments?user.id=98e0eece1e7146c48cf68b599e58c0bf&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=16de5c1385014299a8e7cef6b0114795 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:43.239406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.304233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938cd255-d1c8-45a0-9914-da965522273b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.344213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.345188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.345648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.346075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.347207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.347715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.350808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.358917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 460/904] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:43.365295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.376860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf93b72-e8a6-43b6-856f-1ba911d9a893 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.402791 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:43.410960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:43.427515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.429160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.429649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.430560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.431448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.433173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:43.447574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:43.448589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:43.449947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 445/905] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:43.455286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.515641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1311b4ba-9998-414e-9add-69fbcd1fd9d3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.558267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49137984-63ed-4d2d-aa12-d38dfc8f964d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.563431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.564045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.564325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.564599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.565265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.565568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.567493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.576997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 461/906] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:43.580055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.602991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.603606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.603887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.604240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.604841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.605185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.606551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.616137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 446/907] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:43.617431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.688499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:43.701961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819ca08-fa2c-4fd8-bc2c-a9d1711e4372 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.713304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:43.737692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.738190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.738494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.738767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.739470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] RBAC: Adding query filter params (interface=public, service_id=baa219c2a8c54743b3cbe1ad3bad6eff, region_id=RegionOne) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.739770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.741166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.752959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 462/908] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/endpoints?service_id=baa219c2a8c54743b3cbe1ad3bad6eff&interface=public®ion_id=RegionOne => generated 194 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:43.755706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.758802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b1d0f-9bee-46e8-9c47-1bd53bb0f470 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.804980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.805543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.805813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.806082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.806768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.807070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.808427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.819252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 447/909] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:43.820794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:43.875804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e37329-cc66-4e60-841a-ba3c65bcb24f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.911225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.911877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.912102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.912371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.913009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.914338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.920973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 463/910] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services => generated 1497 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:05:43.936522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efd8c6d-f9c2-445a-8547-2a592e711f83 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:43.964062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:43.975926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ba2dbe-a012-4c0a-811f-59498c41e490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:43.976553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ba2dbe-a012-4c0a-811f-59498c41e490 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:43.976831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ba2dbe-a012-4c0a-811f-59498c41e490 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:43.977138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ba2dbe-a012-4c0a-811f-59498c41e490 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:43.977844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ba2dbe-a012-4c0a-811f-59498c41e490 None admin] RBAC: Adding query filter params (interface=public, service_id=99fc2219bf094e3184decabb76c34343, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:43.978220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ba2dbe-a012-4c0a-811f-59498c41e490 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:43.979596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ba2dbe-a012-4c0a-811f-59498c41e490 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:43.985794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 448/911] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/endpoints?service_id=99fc2219bf094e3184decabb76c34343&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:43.994120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:44.033417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:44.129600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ad55c-d14f-40ba-ba5b-7aacdf6ebb2f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:44.166556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13a9ec-2518-4c8a-bb9d-6e46619c3af9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:44.167153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13a9ec-2518-4c8a-bb9d-6e46619c3af9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:44.167428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13a9ec-2518-4c8a-bb9d-6e46619c3af9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:44.167694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13a9ec-2518-4c8a-bb9d-6e46619c3af9 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:44.168295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba13a9ec-2518-4c8a-bb9d-6e46619c3af9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:44.169667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba13a9ec-2518-4c8a-bb9d-6e46619c3af9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:44.176202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 464/912] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:05:43 2022] GET /identity/v3/services => generated 1497 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:05:44.710392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6769e-fca1-4814-9f55-cc666e6ffcc8 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:44.710903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6769e-fca1-4814-9f55-cc666e6ffcc8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:44.710982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea6769e-fca1-4814-9f55-cc666e6ffcc8 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:44.712381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 449/913] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:44.717915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5efcc4f-4b50-4530-8a61-a43a695f6892 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:44.718428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5efcc4f-4b50-4530-8a61-a43a695f6892 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:44.718930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5efcc4f-4b50-4530-8a61-a43a695f6892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:44.738047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c5efcc4f-4b50-4530-8a61-a43a695f6892 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:44.740086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c5efcc4f-4b50-4530-8a61-a43a695f6892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:44.811128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 465/914] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:45.085909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6769e-fca1-4814-9f55-cc666e6ffcc8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:45.199303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea6769e-fca1-4814-9f55-cc666e6ffcc8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:45.235588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:45.236231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:45.236514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:45.236789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:45.237403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:45.239401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:45.248505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:45.249582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:45.250434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 450/915] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:45 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:45.258001 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5efcc4f-4b50-4530-8a61-a43a695f6892 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:45.275227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:45.368729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5efcc4f-4b50-4530-8a61-a43a695f6892 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:45.404116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5893bceb-310c-4d65-9006-48d19fe2bd25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:45.404730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893bceb-310c-4d65-9006-48d19fe2bd25 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:45.405101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893bceb-310c-4d65-9006-48d19fe2bd25 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:45.405380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893bceb-310c-4d65-9006-48d19fe2bd25 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:45.406013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5893bceb-310c-4d65-9006-48d19fe2bd25 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:45.406317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5893bceb-310c-4d65-9006-48d19fe2bd25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:45.407673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5893bceb-310c-4d65-9006-48d19fe2bd25 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:45.416429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 466/916] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:45 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:45.418871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:45.468652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:45.544326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5386b770-1627-4a98-87d1-ccb6c140b136 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:45.571593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21e2486-aeb1-40d0-852a-50fa7edb0eaa None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:45.571922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21e2486-aeb1-40d0-852a-50fa7edb0eaa None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:45.572147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21e2486-aeb1-40d0-852a-50fa7edb0eaa None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:45.573907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 467/917] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:45.579328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1a8d08-b4f8-4825-b444-0250bfd69d90 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:45.579614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1a8d08-b4f8-4825-b444-0250bfd69d90 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:45.579915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1a8d08-b4f8-4825-b444-0250bfd69d90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:45.587462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:45.588499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:45.589124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:45.589634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:45.590467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:45.590974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:45.592601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:45.599184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 451/918] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:45 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:45.606388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-4b1a8d08-b4f8-4825-b444-0250bfd69d90 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:45.606638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:45.607705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4b1a8d08-b4f8-4825-b444-0250bfd69d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:45.705026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 468/919] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:45.726047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8623ccb4-e61d-4235-b424-2b23c127759e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:45.728484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:45.763192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:45.763818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:45.764100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:45.764425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:45.765156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] RBAC: Adding query filter params (interface=public, service_id=299acbbd59014f90ae8e0bb95ded7f1b, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:45.765467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:45.766827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:45.772542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 452/920] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:05:45 2022] GET /identity/v3/endpoints?service_id=299acbbd59014f90ae8e0bb95ded7f1b&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:45.778167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1a8d08-b4f8-4825-b444-0250bfd69d90 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:45.790266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:45.889896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1a8d08-b4f8-4825-b444-0250bfd69d90 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:45.925215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91b2d8e-ff3a-4e9a-8966-80b0e1044374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:45.925827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91b2d8e-ff3a-4e9a-8966-80b0e1044374 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:45.926775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91b2d8e-ff3a-4e9a-8966-80b0e1044374 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:45.927272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91b2d8e-ff3a-4e9a-8966-80b0e1044374 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:45.927887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91b2d8e-ff3a-4e9a-8966-80b0e1044374 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:45.929282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91b2d8e-ff3a-4e9a-8966-80b0e1044374 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:45.935643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 469/921] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:05:45 2022] GET /identity/v3/services => generated 1497 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:05:46.001943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.126240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc0ff31-fa90-48dc-98c0-58b590224c73 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:46.163047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:46.163694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.164033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.164314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.167981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c43fb-1a6b-4c5c-ba85-4ac6700d2837 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.168268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c43fb-1a6b-4c5c-ba85-4ac6700d2837 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.168538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c43fb-1a6b-4c5c-ba85-4ac6700d2837 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.170025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 470/922] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:46.177326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4142a0-411c-4012-93ce-867726bde0f2 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.177688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:46.178201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4142a0-411c-4012-93ce-867726bde0f2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.178682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4142a0-411c-4012-93ce-867726bde0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:46.182925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:46.183881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:46.184731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 453/923] 10.222.0.26 () {56 vars in 1120 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:46.191298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.203115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8c4142a0-411c-4012-93ce-867726bde0f2 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:46.205150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8c4142a0-411c-4012-93ce-867726bde0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:46.227048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:46.293025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 471/924] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:46.312453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b31a3c-ad96-485e-9db7-45601d8bb01b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:46.352250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207c8ea4-3a88-4b41-bb4d-d0ee485f4080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:46.352883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207c8ea4-3a88-4b41-bb4d-d0ee485f4080 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.353193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207c8ea4-3a88-4b41-bb4d-d0ee485f4080 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.353465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207c8ea4-3a88-4b41-bb4d-d0ee485f4080 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.354243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207c8ea4-3a88-4b41-bb4d-d0ee485f4080 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:46.354551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207c8ea4-3a88-4b41-bb4d-d0ee485f4080 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:46.356110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207c8ea4-3a88-4b41-bb4d-d0ee485f4080 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:46.366830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 454/925] 10.222.0.26 () {56 vars in 1115 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:46.372525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4142a0-411c-4012-93ce-867726bde0f2 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.377320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:46.466996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc841706-bbd5-4750-bdaa-cced651fb183 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.467325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc841706-bbd5-4750-bdaa-cced651fb183 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.467565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc841706-bbd5-4750-bdaa-cced651fb183 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.469176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 455/926] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:46.474925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0e46a9-49e5-43be-b7e2-b1006886640e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.475211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0e46a9-49e5-43be-b7e2-b1006886640e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.475492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0e46a9-49e5-43be-b7e2-b1006886640e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.482620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4142a0-411c-4012-93ce-867726bde0f2 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:46.494608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4e0e46a9-49e5-43be-b7e2-b1006886640e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:46.496007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4e0e46a9-49e5-43be-b7e2-b1006886640e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:46.529793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:46.530675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.531102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.531436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.545763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:46.547975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:46.570340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:46.571348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:46.572246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:46.572246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:46.572246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:46.576441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 472/927] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:46.577347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 456/928] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:46.577487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.589447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0e46a9-49e5-43be-b7e2-b1006886640e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.614702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:46.650369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:46.703329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7acc93-155d-4dcc-8ed1-f6e640052950 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:46.714021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0e46a9-49e5-43be-b7e2-b1006886640e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:46.748425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:46.749201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.749573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.749940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.750676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:46.751092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:46.752429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:46.752884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:46.753097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.753381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.753652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.754481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:46.756052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:46.765541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:46.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 01 04:05:46.768612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:46.770237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 473/929] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:46.776435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 457/930] 10.222.0.26 () {56 vars in 1132 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:46.776663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.777481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.871629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:46.876527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5203297f-db5d-4456-be6b-294e12a9c192 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:46.889392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1f70a5-41f4-4594-a82a-90f800958a19 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:46.893629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:46.916666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:46.917351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.917634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.917912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.925464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:46.925705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:46.927234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:46.927428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:46.927872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:46.928317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:46.929292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:46.930242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:46.931770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:46.937391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 458/931] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:46.938385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:46.938793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:46.939738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:46.940691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 474/932] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:46.942626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.046072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79a716-7862-4247-8988-fb00950398e8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.054257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99209995-dfcc-435a-b22a-9354edd76867 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.084275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.085221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.085703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.085976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.086600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:47.086902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.088244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.090561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.091198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.091485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.091759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.092342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.093772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.094171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 475/933] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:47.094955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:47.107691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:47.108823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 01 04:05:47.109693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:47.113083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 459/934] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:46 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:47.114932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.130430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:47.144148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:47.209772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815823a7-6224-4e49-86bf-23ba3f17a0d7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.227650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40a8724-f295-4b54-9c62-4a3b8a70074c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.248023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.248630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.248905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.249201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.249925 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:47.250228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.251634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.264010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.264886 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.265383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.265991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.266659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.268009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.274102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 476/935] 10.222.0.26 () {56 vars in 1149 bytes} [Sat Oct 1 04:05:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:47.274102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.292878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 460/936] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:05:47 2022] POST /identity/v3/endpoints => generated 347 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:47.293719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.386637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:47.402388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:47.413573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b61fe7-78c1-45fb-ad16-6f296ea3a32f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.447407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3471c67-4e39-44f8-8ed4-c379d385c42b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.472163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.473477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.473755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.474022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.474760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:47.475059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.480740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.487734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 477/937] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:47 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:47.493646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.495533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.496753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.497256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.498708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.512798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.514598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:47.531307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:47.532446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:47.533742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:47.533742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:47.533742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:47.537063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 461/938] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:47.540054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.618222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb5fcb3-6fee-4d42-bfef-04350f8dcfd6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.657168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.657694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.658224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.658497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.659541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:47.659861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.661627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.667535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 478/939] 10.222.0.26 () {56 vars in 1114 bytes} [Sat Oct 1 04:05:47 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:47.673716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.676915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189f49f3-d46a-4845-baa2-29b55d2cd11c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.694476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:47.706476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:47.712868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.713519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.713780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.714048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.714704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:47.714990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.716358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.733072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 462/940] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:47.734684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:47.806480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed90d87-2be6-4cf2-ac3e-1066341cc223 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.842080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929389d0-b3ee-4c7a-b51c-ff6c91af9424 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:47.851041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.851995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.852525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.853020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.853995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.855620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.880199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:47.880840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:47.881144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:47.881412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:47.883158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 479/941] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:05:47 2022] POST /identity/v3/endpoints => generated 346 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:47.896056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:47.897719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:47.912451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:47.913429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:47.914312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 463/942] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:47.919592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:48.016404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:48.045329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b71ce5-f92f-4f9a-a630-ea0bbbadb30b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:48.082811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10a1bc6-49d2-4915-b7d4-b1588d3d1261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:48.083422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a1bc6-49d2-4915-b7d4-b1588d3d1261 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:48.083747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a1bc6-49d2-4915-b7d4-b1588d3d1261 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:48.084023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10a1bc6-49d2-4915-b7d4-b1588d3d1261 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:48.084776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10a1bc6-49d2-4915-b7d4-b1588d3d1261 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:48.085045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10a1bc6-49d2-4915-b7d4-b1588d3d1261 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:48.086454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10a1bc6-49d2-4915-b7d4-b1588d3d1261 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:48.107062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:48.107526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 480/943] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:48.149175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:48.220139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eb05eb-9302-4726-b981-f26e42ae8193 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:48.232736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd3896-afef-4b59-b3c9-c1c17aa5a006 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:48.233114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd3896-afef-4b59-b3c9-c1c17aa5a006 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:48.233395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfd3896-afef-4b59-b3c9-c1c17aa5a006 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:48.235621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 481/944] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:48.241733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb5e03-5fc8-4427-8ed9-8361070e5e6f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:48.242024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb5e03-5fc8-4427-8ed9-8361070e5e6f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:48.242299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb5e03-5fc8-4427-8ed9-8361070e5e6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:48.260347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:48.261127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:48.261423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:48.261700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/98e0eece1e7146c48cf68b599e58c0bf/roles/16de5c1385014299a8e7cef6b0114795` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:48.261996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-27fb5e03-5fc8-4427-8ed9-8361070e5e6f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:48.263197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-27fb5e03-5fc8-4427-8ed9-8361070e5e6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:48.263889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=98e0eece1e7146c48cf68b599e58c0bf, role_id=16de5c1385014299a8e7cef6b0114795)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:48.265669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:48.280984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 464/945] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:05:48 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/98e0eece1e7146c48cf68b599e58c0bf/roles/16de5c1385014299a8e7cef6b0114795 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:05:48.343154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 482/946] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:48.616764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:48.717423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:48.727359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761f272e-c89f-4849-ad95-66209985a883 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:48.764041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:48.765056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:48.765644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:48.766154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:48.766929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:48.768490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:48.778195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:48.779196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:48.780039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:48.782838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 465/947] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:48.789273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb5e03-5fc8-4427-8ed9-8361070e5e6f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:48.862558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:48.899452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb5e03-5fc8-4427-8ed9-8361070e5e6f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:48.937795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:48.938874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:48.939353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:48.939840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:48.940689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:48.941243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:48.943497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:48.949603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 483/948] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:48 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:48.958192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:49.062225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ff41d-3d42-49a8-a9fe-f70002032797 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:49.102421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3421f7-0b9c-4b29-9783-237a36a760b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:49.103377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3421f7-0b9c-4b29-9783-237a36a760b7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:49.103874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3421f7-0b9c-4b29-9783-237a36a760b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:49.104359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3421f7-0b9c-4b29-9783-237a36a760b7 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:49.105200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3421f7-0b9c-4b29-9783-237a36a760b7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:49.105714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3421f7-0b9c-4b29-9783-237a36a760b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:49.107252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3421f7-0b9c-4b29-9783-237a36a760b7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:49.112951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 466/949] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:48 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:49.119074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:49.225074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3524f8-7d01-4a50-beed-7149d684736f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:49.255159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:49.261951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b711f03-79ce-45ee-ba58-2d0ce0a49884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:49.263030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b711f03-79ce-45ee-ba58-2d0ce0a49884 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:49.263508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b711f03-79ce-45ee-ba58-2d0ce0a49884 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:49.263973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b711f03-79ce-45ee-ba58-2d0ce0a49884 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:49.264842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b711f03-79ce-45ee-ba58-2d0ce0a49884 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:49.266403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b711f03-79ce-45ee-ba58-2d0ce0a49884 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:49.288481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 484/950] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:05:49 2022] POST /identity/v3/endpoints => generated 349 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:49.679367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96789a72-d124-47d3-a0b8-2f0b8710d663 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:49.680125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96789a72-d124-47d3-a0b8-2f0b8710d663 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:49.680659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96789a72-d124-47d3-a0b8-2f0b8710d663 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:49.682481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 467/951] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:49.687918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f32b0-9d91-4b5b-b9ba-4ae2123ae77a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:49.688431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f32b0-9d91-4b5b-b9ba-4ae2123ae77a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:49.688908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f32b0-9d91-4b5b-b9ba-4ae2123ae77a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:49.708230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-860f32b0-9d91-4b5b-b9ba-4ae2123ae77a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:49.709730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-860f32b0-9d91-4b5b-b9ba-4ae2123ae77a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:49.784765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 485/952] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:50.059782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96789a72-d124-47d3-a0b8-2f0b8710d663 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:50.088700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:50.182342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96789a72-d124-47d3-a0b8-2f0b8710d663 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:50.226701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae27ae-acb5-4dd3-ae97-9ee84de7ffb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:50.227358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae27ae-acb5-4dd3-ae97-9ee84de7ffb7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.227636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae27ae-acb5-4dd3-ae97-9ee84de7ffb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.227904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae27ae-acb5-4dd3-ae97-9ee84de7ffb7 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.228486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae27ae-acb5-4dd3-ae97-9ee84de7ffb7 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:50.229952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae27ae-acb5-4dd3-ae97-9ee84de7ffb7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f9ae27ae-acb5-4dd3-ae97-9ee84de7ffb7 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:50.239220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:50.240351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:50.241202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 468/953] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:50 2022] GET /identity/v3/services/compute => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:50.248766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860f32b0-9d91-4b5b-b9ba-4ae2123ae77a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:50.321031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6e52f-ad1e-41b4-b805-c08e9f6a40a4 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.321207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6e52f-ad1e-41b4-b805-c08e9f6a40a4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.321484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6e52f-ad1e-41b4-b805-c08e9f6a40a4 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.322930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 469/954] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:50.328088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f38623-6fa7-4749-a250-9c0921abb9f3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.328368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f38623-6fa7-4749-a250-9c0921abb9f3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.328638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f38623-6fa7-4749-a250-9c0921abb9f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.346781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a8f38623-6fa7-4749-a250-9c0921abb9f3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:50.348671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a8f38623-6fa7-4749-a250-9c0921abb9f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:50.359156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860f32b0-9d91-4b5b-b9ba-4ae2123ae77a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:50.397362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:50.402914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:50.403927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.404213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.404488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.405150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] RBAC: Adding query filter params (name=compute) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:50.405454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:50.407422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:50.414347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 486/955] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:50 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:50.419976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:50.427197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 470/956] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:50.537495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad98c02b-6679-4ad6-9b8d-8a84607fedaf None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:50.560144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674535aa-4d9d-429d-bfe3-3d66c64a08e9 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.560451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674535aa-4d9d-429d-bfe3-3d66c64a08e9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.560707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674535aa-4d9d-429d-bfe3-3d66c64a08e9 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.562311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 471/957] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:50.568174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b18884-d25a-4171-921d-dd7f34dd86ec None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.568459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b18884-d25a-4171-921d-dd7f34dd86ec None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.568738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b18884-d25a-4171-921d-dd7f34dd86ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.574845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:50.575660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.575937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.577134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.577774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:50.578078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:50.579416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:50.585484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 487/958] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:50 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:50.587817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f9b18884-d25a-4171-921d-dd7f34dd86ec None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:50.592987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f9b18884-d25a-4171-921d-dd7f34dd86ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:50.604738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:50.668833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 472/959] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:50.720008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:50.767729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:50.827409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d59d859-547d-45ee-8706-5870ea896cd6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:50.862331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:50.862941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:50.863315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:50.863589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:50.864231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:50.865667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:50.868242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING keystone.server.flask.common [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 01 04:05:50.886225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.sql.core [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 01 04:05:50.886225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 01 04:05:50.886225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [parameters: {'id': 'c35e073f4b1940c8aaab132c42aced39', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 01 04:05:50.886225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81586) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c35e073f4b1940c8aaab132c42aced39.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c35e073f4b1940c8aaab132c42aced39. Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result = self._query(query) Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application conn.query(q) Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 01 04:05:50.893889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result.read() Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 01 04:05:50.895424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application next(self.gen) Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application yield resource Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application next(self.gen) Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.session.rollback() Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.force_reraise() Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise self.value Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application session.commit() Oct 01 04:05:50.896409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._prepare_impl() Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.session.flush() Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._flush(objects) Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application compat.raise_( Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application flush_context.execute() Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 01 04:05:50.897493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rec.execute(self) Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 01 04:05:50.898420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result = self._query(query) Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application conn.query(q) Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result.read() Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 01 04:05:50.899209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [parameters: {'id': 'c35e073f4b1940c8aaab132c42aced39', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:50.900094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:50.900886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id c35e073f4b1940c8aaab132c42aced39. Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:50.906279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 488/960] 10.222.0.26 () {60 vars in 1140 bytes} [Sat Oct 1 04:05:50 2022] POST /identity/v3/projects => generated 262 bytes in 177 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:50.906885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b18884-d25a-4171-921d-dd7f34dd86ec None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:50.964331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:51.018818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b18884-d25a-4171-921d-dd7f34dd86ec None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:51.063850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:51.065830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.066867 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.067700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.073389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:51.077198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:51.078832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:51.086141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052da5da-f114-4c26-9a6d-3ed5efb8a006 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:51.097660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:51.098674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:51.099694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:51.108606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 473/961] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:51.110527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:51.126173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:51.131710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:51.132451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.132849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.133253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.154235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:05:51.159769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:51.161262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:51.162128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 489/962] 10.222.0.26 () {56 vars in 1120 bytes} [Sat Oct 1 04:05:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:51.167589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:51.241135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c36879-7dd6-43f9-8fae-4f52b0748591 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:51.283573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:51.284531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.285068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.285943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.286958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:51.287467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:51.289110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:51.289402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c092ba48-36ba-42f5-b345-4275bf547e27 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:51.308963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 474/963] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:51 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:51.340192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:51.341211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.341713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.342186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.343083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:51.343707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:51.345619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:51.359027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 490/964] 10.222.0.26 () {56 vars in 1115 bytes} [Sat Oct 1 04:05:51 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:51.365621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:51.444068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:51.484571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160ee626-29d9-4f76-b9a5-0d199b855330 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:51.528522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704ee00f-38d2-4a22-82b2-44c99beaeb03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:51.529221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ee00f-38d2-4a22-82b2-44c99beaeb03 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.529503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ee00f-38d2-4a22-82b2-44c99beaeb03 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.529781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ee00f-38d2-4a22-82b2-44c99beaeb03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.540726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704ee00f-38d2-4a22-82b2-44c99beaeb03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:51.542202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704ee00f-38d2-4a22-82b2-44c99beaeb03 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-704ee00f-38d2-4a22-82b2-44c99beaeb03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:51.557439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:51.558452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:51.559354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:51.559354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:51.559354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:51.560336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 475/965] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:51.564641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:51.589732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:51.609584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dab8f2-3a69-4516-a1af-49710b4e2621 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.609861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dab8f2-3a69-4516-a1af-49710b4e2621 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.610044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dab8f2-3a69-4516-a1af-49710b4e2621 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.611608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 476/966] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:51.618288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fd6441-46a3-4d25-a07d-ec179503aa9a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.618574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fd6441-46a3-4d25-a07d-ec179503aa9a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.618836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fd6441-46a3-4d25-a07d-ec179503aa9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.638580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-57fd6441-46a3-4d25-a07d-ec179503aa9a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:51.639773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-57fd6441-46a3-4d25-a07d-ec179503aa9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:51.687569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3405f0e-f785-4299-97c7-a87251954d3d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:51.729026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 477/967] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:51.732499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:51.733151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.733438 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.733717 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.734368 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:51.734675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:51.736093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:51.749824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 491/968] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:51.758006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fd6441-46a3-4d25-a07d-ec179503aa9a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:51.810530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:51.865774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fd6441-46a3-4d25-a07d-ec179503aa9a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:51.901693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:51.902729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:51.903211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:51.903693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:51.911735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:51.913529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:51.926105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:51.927063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 01 04:05:51.928056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:51.935191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 478/969] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:05:51 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:51.938019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.029915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:52.046351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.047703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa0ae46-7740-4593-9d22-7555038725dd None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.088129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.088733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.089049 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.089328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.090057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:52.090366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.091803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.115136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 492/970] 10.222.0.26 () {56 vars in 1149 bytes} [Sat Oct 1 04:05:51 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:52.117246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.162996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3cadbe-27b7-4ab2-8958-69e4c1c4e705 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.174364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:52.201930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.202965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.203483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.203957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.204738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.206324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:52.216405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:52.217653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:52.218514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:52.226077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 479/971] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/services/object-store => generated 93 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:52.227569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.229099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faabfad-d7a1-4eca-8819-f16d9a01f597 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.267065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.267679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.267960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.268234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.272083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:52.337566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b5f80f-7fb4-4617-b3a8-5dd985ab2809 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.354490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.357029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.375195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.375993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.376264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.376649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.377446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:52.377860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.379910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.390957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 480/972] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:52.392369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.436333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:52.476573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 493/973] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:05:52.481922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.510260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7bfb93-22c4-4bed-8212-ec3f722de306 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.521244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:52.552954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.553847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.554110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.554450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.555143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:52.555512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.556949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.567906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 481/974] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:52.569405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.595913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462fe676-76d7-41da-925c-1487777ecf83 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.642106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.642686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.642991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.643285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.655665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.657396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:05:52.674549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:52.676120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:05:52.677060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:05:52.677060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:05:52.677060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:52.681068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 494/975] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:52.682481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.694169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84708c0b-6a63-4d48-bb4c-8f0512742623 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.696148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:52.732232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.733356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.733852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.734341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.735322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] RBAC: Adding query filter params (interface=admin, service_id=299acbbd59014f90ae8e0bb95ded7f1b, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:52.735855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.737548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.749077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 482/976] 10.222.0.26 () {56 vars in 1249 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/endpoints?service_id=299acbbd59014f90ae8e0bb95ded7f1b&interface=admin®ion_id=RegionOne => generated 193 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:52.750813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.804846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:52.821155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2f779c-2d19-4441-ab7b-85c18453ac18 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.863696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc71913-eee5-42cd-95a4-d181dd2c79ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.863938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4a592-ab02-4ce0-9541-6f52c98cbcd9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:52.864318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc71913-eee5-42cd-95a4-d181dd2c79ef None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.864598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc71913-eee5-42cd-95a4-d181dd2c79ef None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.864873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc71913-eee5-42cd-95a4-d181dd2c79ef None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.865553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc71913-eee5-42cd-95a4-d181dd2c79ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:52.865858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc71913-eee5-42cd-95a4-d181dd2c79ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.867226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc71913-eee5-42cd-95a4-d181dd2c79ef None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.885299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 495/977] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:52.886577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91dd2ce-319d-4ea3-99c5-73a715ad52f3 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.886868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91dd2ce-319d-4ea3-99c5-73a715ad52f3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.887140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91dd2ce-319d-4ea3-99c5-73a715ad52f3 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.888501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 496/978] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:52.889555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91dd2ce-319d-4ea3-99c5-73a715ad52f3 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:52.903743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3baa339f-665f-49df-976a-0c712eab7ef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:52.904284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa339f-665f-49df-976a-0c712eab7ef2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.904589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa339f-665f-49df-976a-0c712eab7ef2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.904836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3baa339f-665f-49df-976a-0c712eab7ef2 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.905468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baa339f-665f-49df-976a-0c712eab7ef2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:52.906835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3baa339f-665f-49df-976a-0c712eab7ef2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:52.916573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 483/979] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/services => generated 1497 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:05:52.916573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73da458f-5916-4ee5-ac2b-e160e3a45c88 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:52.916573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73da458f-5916-4ee5-ac2b-e160e3a45c88 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:52.916573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73da458f-5916-4ee5-ac2b-e160e3a45c88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:52.939837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-73da458f-5916-4ee5-ac2b-e160e3a45c88 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:52.942259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73da458f-5916-4ee5-ac2b-e160e3a45c88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:52.970943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:53.008812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91dd2ce-319d-4ea3-99c5-73a715ad52f3 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:53.028233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 484/980] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:53.048543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:53.049216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:53.049493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:53.049769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:53.057533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:53.059089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:05:53.072962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:53.074155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:05:53.075126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:53.079088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 497/981] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:53.080581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73da458f-5916-4ee5-ac2b-e160e3a45c88 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:53.195086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73da458f-5916-4ee5-ac2b-e160e3a45c88 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:53.234973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:53.235596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:53.235879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:53.236154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:53.236851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:53.237186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:53.239427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:53.256753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 485/982] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:05:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:53.258166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:53.269644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:53.349758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:53.370383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f7dad8-9f16-48b7-8960-095e2152e559 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:53.377924 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:53.415383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461560c4-a462-438d-a17b-9ee6ebf1a848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:53.415991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461560c4-a462-438d-a17b-9ee6ebf1a848 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:53.416272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461560c4-a462-438d-a17b-9ee6ebf1a848 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:53.416539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461560c4-a462-438d-a17b-9ee6ebf1a848 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:53.417292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461560c4-a462-438d-a17b-9ee6ebf1a848 None admin] RBAC: Adding query filter params (role.id=16de5c1385014299a8e7cef6b0114795, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=98e0eece1e7146c48cf68b599e58c0bf) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:53.417602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461560c4-a462-438d-a17b-9ee6ebf1a848 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:53.419325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461560c4-a462-438d-a17b-9ee6ebf1a848 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:53.433023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 498/983] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:05:53 2022] GET /identity/v3/role_assignments?user.id=98e0eece1e7146c48cf68b599e58c0bf&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=16de5c1385014299a8e7cef6b0114795 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:53.487484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62817c80-cf60-49f2-be43-a2d0e20be2cc None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:53.533178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c638c8-8ff2-4476-925a-1d1fca47e088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:53.534230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c638c8-8ff2-4476-925a-1d1fca47e088 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:53.534765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c638c8-8ff2-4476-925a-1d1fca47e088 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:53.535291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c638c8-8ff2-4476-925a-1d1fca47e088 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:53.536284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c638c8-8ff2-4476-925a-1d1fca47e088 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:53.537956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c638c8-8ff2-4476-925a-1d1fca47e088 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:53.557547 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 486/984] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:53 2022] POST /identity/v3/services => generated 242 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:05:53.712634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288c42be-adc5-4ce6-b841-ebe4f4c00aa2 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:53.712994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288c42be-adc5-4ce6-b841-ebe4f4c00aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:53.713231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288c42be-adc5-4ce6-b841-ebe4f4c00aa2 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:53.714782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 499/985] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:53.720973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f7550-929d-469b-bfd4-4612a0a2c669 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:53.731975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f7550-929d-469b-bfd4-4612a0a2c669 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:53.731975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f7550-929d-469b-bfd4-4612a0a2c669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:53.744328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b80f7550-929d-469b-bfd4-4612a0a2c669 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:53.745940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b80f7550-929d-469b-bfd4-4612a0a2c669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:53.832710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 487/986] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:53 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:54.107584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288c42be-adc5-4ce6-b841-ebe4f4c00aa2 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:54.183416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:54.216102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288c42be-adc5-4ce6-b841-ebe4f4c00aa2 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:54.250411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:54.251021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:54.251304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:54.251578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:54.252239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:54.253773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:54.276953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.sql.core [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 01 04:05:54.276953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 01 04:05:54.276953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [parameters: {'user_id': '5ea841d4bb8e4f8293abd7547d3039f3', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 01 04:05:54.276953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81586) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result = self._query(query) Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application conn.query(q) Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result.read() Oct 01 04:05:54.279192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 01 04:05:54.280105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application next(self.gen) Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application yield resource Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application next(self.gen) Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.session.rollback() Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.force_reraise() Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise self.value Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application session.commit() Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 01 04:05:54.281149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._prepare_impl() Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.session.flush() Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._flush(objects) Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application compat.raise_( Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application flush_context.execute() Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rec.execute(self) Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 01 04:05:54.282065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 01 04:05:54.283285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result = self._query(query) Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application conn.query(q) Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application result.read() Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 01 04:05:54.284188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [parameters: {'user_id': '5ea841d4bb8e4f8293abd7547d3039f3', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:54.286861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 01 04:05:54.288776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 01 04:05:54.292709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 01 04:05:54.292709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 01 04:05:54.292709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 01 04:05:54.292709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 01 04:05:54.292709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 01 04:05:54.292709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:54.292709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 500/987] 10.222.0.26 () {60 vars in 1131 bytes} [Sat Oct 1 04:05:54 2022] POST /identity/v3/users => generated 160 bytes in 185 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:54.301028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80f7550-929d-469b-bfd4-4612a0a2c669 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:54.314972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:54.417663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80f7550-929d-469b-bfd4-4612a0a2c669 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:54.453706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648a0d27-738c-4196-a8da-a3615dc1e64b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:54.454370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648a0d27-738c-4196-a8da-a3615dc1e64b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:54.454651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648a0d27-738c-4196-a8da-a3615dc1e64b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:54.454922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648a0d27-738c-4196-a8da-a3615dc1e64b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:54.463906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648a0d27-738c-4196-a8da-a3615dc1e64b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:54.465410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648a0d27-738c-4196-a8da-a3615dc1e64b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-648a0d27-738c-4196-a8da-a3615dc1e64b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:05:54.478025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:54.479045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:54.480064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 488/988] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:05:54 2022] GET /identity/v3/users/cinder => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:54.486927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:54.600455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7350f508-6f98-4997-b579-db7e8df0549f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:54.635319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefd3139-238f-4fc7-a097-cd0c1dea6ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:54.636256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefd3139-238f-4fc7-a097-cd0c1dea6ee9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:54.636753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefd3139-238f-4fc7-a097-cd0c1dea6ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:54.637275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefd3139-238f-4fc7-a097-cd0c1dea6ee9 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:54.638176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefd3139-238f-4fc7-a097-cd0c1dea6ee9 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:54.638702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefd3139-238f-4fc7-a097-cd0c1dea6ee9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:54.640586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefd3139-238f-4fc7-a097-cd0c1dea6ee9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:54.642484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:54.683272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 501/989] 10.222.0.26 () {56 vars in 1101 bytes} [Sat Oct 1 04:05:54 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:55.181299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd555fd1-ff16-434f-999e-99d462207b03 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.181618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd555fd1-ff16-434f-999e-99d462207b03 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.181854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd555fd1-ff16-434f-999e-99d462207b03 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.183322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 489/990] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:55.188410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad84006e-c89e-4d48-ab56-0dcd2f2a56a1 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.188699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad84006e-c89e-4d48-ab56-0dcd2f2a56a1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.188998 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad84006e-c89e-4d48-ab56-0dcd2f2a56a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.207943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ad84006e-c89e-4d48-ab56-0dcd2f2a56a1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:55.209445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ad84006e-c89e-4d48-ab56-0dcd2f2a56a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:55.275594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 502/991] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:55 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:55.547965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd555fd1-ff16-434f-999e-99d462207b03 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:55.615150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:55.652398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd555fd1-ff16-434f-999e-99d462207b03 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:55.687640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d16b59f-8b86-468a-a798-14bd9f0b5d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:55.688366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d16b59f-8b86-468a-a798-14bd9f0b5d5f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.688643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d16b59f-8b86-468a-a798-14bd9f0b5d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.689004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d16b59f-8b86-468a-a798-14bd9f0b5d5f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.689644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d16b59f-8b86-468a-a798-14bd9f0b5d5f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:55.691082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d16b59f-8b86-468a-a798-14bd9f0b5d5f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3d16b59f-8b86-468a-a798-14bd9f0b5d5f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:55.701476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:55.702591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 01 04:05:55.703471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:55.705132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 490/992] 10.222.0.26 () {56 vars in 1126 bytes} [Sat Oct 1 04:05:55 2022] GET /identity/v3/services/object-store => generated 93 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:55.708875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad84006e-c89e-4d48-ab56-0dcd2f2a56a1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:55.812982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbfac5a-4f20-46af-974a-c856b507a40b None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.813177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbfac5a-4f20-46af-974a-c856b507a40b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.813433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbfac5a-4f20-46af-974a-c856b507a40b None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.814259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad84006e-c89e-4d48-ab56-0dcd2f2a56a1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:55.814881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 491/993] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:55.820292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d075de-4bfc-4e03-b6c7-7e75630f3f02 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.820576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d075de-4bfc-4e03-b6c7-7e75630f3f02 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.820849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d075de-4bfc-4e03-b6c7-7e75630f3f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.838384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:55.839772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-91d075de-4bfc-4e03-b6c7-7e75630f3f02 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:55.841044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-91d075de-4bfc-4e03-b6c7-7e75630f3f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:55.852546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2827d5-bdf9-4c39-9920-7442ef2c401e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:55.853378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2827d5-bdf9-4c39-9920-7442ef2c401e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.853825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2827d5-bdf9-4c39-9920-7442ef2c401e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.854244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2827d5-bdf9-4c39-9920-7442ef2c401e None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.854872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2827d5-bdf9-4c39-9920-7442ef2c401e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:55.855172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2827d5-bdf9-4c39-9920-7442ef2c401e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:55.857152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2827d5-bdf9-4c39-9920-7442ef2c401e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:55.862846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 503/994] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:55 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:55.864199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608be915-5338-410b-b8ed-4c0ba203f2f8 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.864480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608be915-5338-410b-b8ed-4c0ba203f2f8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.864750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608be915-5338-410b-b8ed-4c0ba203f2f8 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.866102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 504/995] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:55.871178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ca8be3-a8c6-41d2-be81-db9fd2ade02a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:55.871466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ca8be3-a8c6-41d2-be81-db9fd2ade02a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:55.871746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ca8be3-a8c6-41d2-be81-db9fd2ade02a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:55.889650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-64ca8be3-a8c6-41d2-be81-db9fd2ade02a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:55.890841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-64ca8be3-a8c6-41d2-be81-db9fd2ade02a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:55.914752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 492/996] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:55 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:55.915591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d075de-4bfc-4e03-b6c7-7e75630f3f02 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:55.964120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 505/997] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:55 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:55.979494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:56.022128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d075de-4bfc-4e03-b6c7-7e75630f3f02 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.056689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.057323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.057606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.057879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.058498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:56.058803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.060160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.065254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 493/998] 10.222.0.26 () {56 vars in 1122 bytes} [Sat Oct 1 04:05:55 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:56.071673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ca8be3-a8c6-41d2-be81-db9fd2ade02a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.174591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ca8be3-a8c6-41d2-be81-db9fd2ade02a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.189515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.191985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:56.214353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.214963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.215246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.215522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.216202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.217595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.234252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 506/999] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:05:56 2022] POST /identity/v3/endpoints => generated 325 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:05:56.241021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.316826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02f96ad-658b-4e95-b598-2dd950542e01 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.342017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:56.360654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.361327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.361602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.361871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] PATH_INFO: `/v3/services/image` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.362457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.364460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.372279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2053def8-0439-42cb-8e2b-7ceeafe98369 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:56.373624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:56.381133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:56.382088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 494/1000] 10.222.0.26 () {56 vars in 1105 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/services/image => generated 86 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:56.385515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.416872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.417827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.418328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.418607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.419218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.420702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:56.431838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:56.432969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:05:56.433999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 507/1001] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/services/compute => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:05:56.438779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.489279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:56.509207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25af53d3-39d9-49c4-a398-5c064286c55e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.546477 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.547082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.547358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.547625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.548245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] RBAC: Adding query filter params (name=image) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:56.548777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.549916 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.555627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 495/1002] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/services?name=image => generated 122 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:56.560910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.563390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ae772a-6f31-4a15-8fb9-cee5ddb0b002 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.600459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.600999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.601279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.601546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.602163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:56.602467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.603798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.609189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 508/1003] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:56.612464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:56.615423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.666520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a8f138-1f52-48ad-8329-59fab12f6c8c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.699986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.700690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.701068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.701411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.702107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] RBAC: Adding query filter params (type=image) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:56.702489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.703913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.720960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 496/1004] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/services?type=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:56.736489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630c351d-ebde-40a7-8753-4f019c8161f2 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.739384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:56.775320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.775949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.776232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.776504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.777188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:56.777495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.778871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.785590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 509/1005] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:05:56.791495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:56.904739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f4f03-1ed4-4b2c-a6a1-91d2963e32de None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:56.918179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:56.940244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae683833-0221-42ef-aab5-a4ccb1748217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:56.941594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae683833-0221-42ef-aab5-a4ccb1748217 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:56.942057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae683833-0221-42ef-aab5-a4ccb1748217 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:56.942445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae683833-0221-42ef-aab5-a4ccb1748217 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:56.943246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae683833-0221-42ef-aab5-a4ccb1748217 None admin] RBAC: Adding query filter params (interface=public, service_id=8c4b44bc6382487b87270c95442cafe3, region_id=RegionOne) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:05:56.943655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae683833-0221-42ef-aab5-a4ccb1748217 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:56.945135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae683833-0221-42ef-aab5-a4ccb1748217 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:56.950603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 497/1006] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/endpoints?service_id=8c4b44bc6382487b87270c95442cafe3&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:56.956200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:57.059206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c80b716-48e4-4754-9fc9-d7874f303748 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:57.093850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4092886f-c540-4360-8b31-25e82d3afd5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:57.094505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4092886f-c540-4360-8b31-25e82d3afd5e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:57.094788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4092886f-c540-4360-8b31-25e82d3afd5e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:57.095060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4092886f-c540-4360-8b31-25e82d3afd5e None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:57.095663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4092886f-c540-4360-8b31-25e82d3afd5e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:57.097034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4092886f-c540-4360-8b31-25e82d3afd5e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:57.113600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 510/1007] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:05:56 2022] GET /identity/v3/services => generated 1727 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:05:58.440513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cc7b1-cef7-4289-b572-384ca2168465 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:58.441721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cc7b1-cef7-4289-b572-384ca2168465 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:58.441721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27cc7b1-cef7-4289-b572-384ca2168465 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:58.443200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 498/1008] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:05:58.448515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f8b1f3-0879-4395-9284-18b0dab608ef None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:58.448980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f8b1f3-0879-4395-9284-18b0dab608ef None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:58.449254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f8b1f3-0879-4395-9284-18b0dab608ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:58.467318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-45f8b1f3-0879-4395-9284-18b0dab608ef None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:58.468615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-45f8b1f3-0879-4395-9284-18b0dab608ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:58.497733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:58.546869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 511/1009] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:58 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:05:58.818966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27cc7b1-cef7-4289-b572-384ca2168465 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:58.897416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df53ba9-2c97-4c1c-ba7f-7df62cb48b68 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:58.897761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df53ba9-2c97-4c1c-ba7f-7df62cb48b68 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:58.897954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df53ba9-2c97-4c1c-ba7f-7df62cb48b68 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:58.899432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 512/1010] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:58.904354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4cea1e-848d-41bb-85ba-59658d7b7d04 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:58.904638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4cea1e-848d-41bb-85ba-59658d7b7d04 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:58.904912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4cea1e-848d-41bb-85ba-59658d7b7d04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:58.924869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0a4cea1e-848d-41bb-85ba-59658d7b7d04 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:58.926390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0a4cea1e-848d-41bb-85ba-59658d7b7d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:58.927494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27cc7b1-cef7-4289-b572-384ca2168465 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:58.966223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:58.967252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:58.967543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:58.967819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:58.969367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:58.971387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:58.973389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 499/1011] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:05:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:05:58.978764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:59.004972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 513/1012] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:58 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:59.030571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:59.098047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d4ca69-40c2-4c65-84d5-73f9afde5923 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:59.137714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff2070a-907f-41ee-8a59-c28ae8b1c206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:59.138338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2070a-907f-41ee-8a59-c28ae8b1c206 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:59.138622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2070a-907f-41ee-8a59-c28ae8b1c206 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:59.138893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff2070a-907f-41ee-8a59-c28ae8b1c206 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:59.139563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff2070a-907f-41ee-8a59-c28ae8b1c206 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:59.141027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff2070a-907f-41ee-8a59-c28ae8b1c206 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:59.185171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 500/1013] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:58 2022] POST /identity/v3/projects => generated 311 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:05:59.284020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4cea1e-848d-41bb-85ba-59658d7b7d04 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:59.285800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4678d-3916-4caa-8bcb-6d4ee1e6c07c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:59.286129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4678d-3916-4caa-8bcb-6d4ee1e6c07c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:59.286406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4678d-3916-4caa-8bcb-6d4ee1e6c07c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:59.288161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 501/1014] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:05:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:05:59.293533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4d0d7-f701-4ad1-a62c-84e1fd77ff3b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:59.293838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4d0d7-f701-4ad1-a62c-84e1fd77ff3b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:59.294142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4d0d7-f701-4ad1-a62c-84e1fd77ff3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:59.318124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c1e4d0d7-f701-4ad1-a62c-84e1fd77ff3b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:05:59.318124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c1e4d0d7-f701-4ad1-a62c-84e1fd77ff3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:05:59.413607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:59.413788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 502/1015] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:05:59 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:05:59.426067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4cea1e-848d-41bb-85ba-59658d7b7d04 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:59.469141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:59.470409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:59.470853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:59.471694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:59.472713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:59.474385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:59.498824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 514/1016] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:05:59 2022] POST /identity/v3/services => generated 242 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:05:59.690051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4d0d7-f701-4ad1-a62c-84e1fd77ff3b None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:59.718106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:05:59.796710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4d0d7-f701-4ad1-a62c-84e1fd77ff3b None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:59.838896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:05:59.839573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:05:59.839856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:05:59.840131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] PATH_INFO: `/v3/services/compute` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:05:59.840714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:05:59.842109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:05:59.851911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:05:59.853202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 01 04:05:59.854541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:05:59.855551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 503/1017] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:05:59 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:05:59.859103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:05:59.965255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b452ab04-aa85-4b80-8978-b28d57d4b28e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:05:59.999892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:00.000500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:00.000779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:00.001081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:00.001710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:00.002010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:00.005110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:00.014217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 515/1018] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:05:59 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:00.015787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:00.105625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:00.121915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0e182f-0d72-4f46-b05f-8f51dedef3ef None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:00.154973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e8482c-ca3c-4f77-8e8e-c06901ce071a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:00.155589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e8482c-ca3c-4f77-8e8e-c06901ce071a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:00.155872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e8482c-ca3c-4f77-8e8e-c06901ce071a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:00.156145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e8482c-ca3c-4f77-8e8e-c06901ce071a None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:00.156773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e8482c-ca3c-4f77-8e8e-c06901ce071a None admin] RBAC: Adding query filter params (type=compute) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:00.157112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e8482c-ca3c-4f77-8e8e-c06901ce071a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:00.158738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e8482c-ca3c-4f77-8e8e-c06901ce071a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:00.169249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 504/1019] 10.222.0.26 () {56 vars in 1112 bytes} [Sat Oct 1 04:06:00 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:00.170661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:00.190960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:00.277484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c61f5ac-7295-46c4-a0d8-a94a3332b128 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:00.312713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7552c6c9-8d15-431f-b712-9b32ec1c4ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:00.313350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7552c6c9-8d15-431f-b712-9b32ec1c4ec9 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:00.313627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7552c6c9-8d15-431f-b712-9b32ec1c4ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:00.313895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7552c6c9-8d15-431f-b712-9b32ec1c4ec9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:00.314549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7552c6c9-8d15-431f-b712-9b32ec1c4ec9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:00.315948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7552c6c9-8d15-431f-b712-9b32ec1c4ec9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:00.336333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 516/1020] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:06:00 2022] POST /identity/v3/endpoints => generated 334 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:06:01.461886 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00ff433-be5c-4cb4-9a21-d9f2929e51d5 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:01.462409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00ff433-be5c-4cb4-9a21-d9f2929e51d5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:01.462409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00ff433-be5c-4cb4-9a21-d9f2929e51d5 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:01.463907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 505/1021] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:01.468981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c4ff43-4bf8-45a9-8127-9c12b806399a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:01.469413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c4ff43-4bf8-45a9-8127-9c12b806399a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:01.469556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c4ff43-4bf8-45a9-8127-9c12b806399a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:01.488737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c7c4ff43-4bf8-45a9-8127-9c12b806399a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:01.489999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c7c4ff43-4bf8-45a9-8127-9c12b806399a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:01.564762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:01.576999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 517/1022] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:01.711369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9334741a-95a4-47f1-a44a-2913c8451db7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:01.711687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9334741a-95a4-47f1-a44a-2913c8451db7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:01.711982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9334741a-95a4-47f1-a44a-2913c8451db7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:01.713468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 506/1023] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:01.718490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84594cf-9432-4e30-be87-45f8c797b3bb None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:01.718782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84594cf-9432-4e30-be87-45f8c797b3bb None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:01.719058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84594cf-9432-4e30-be87-45f8c797b3bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:01.737146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a84594cf-9432-4e30-be87-45f8c797b3bb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:01.738886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a84594cf-9432-4e30-be87-45f8c797b3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:01.807204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 518/1024] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:01.844054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9334741a-95a4-47f1-a44a-2913c8451db7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:01.948718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9334741a-95a4-47f1-a44a-2913c8451db7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:01.983182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:01.984132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:01.984606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:01.985108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:01.986407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:01.988615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:01.990720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 507/1025] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:01.996104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84594cf-9432-4e30-be87-45f8c797b3bb None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:02.075514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:02.094748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:02.105591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84594cf-9432-4e30-be87-45f8c797b3bb None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:02.134223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:02.144265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:02.145067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.145349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.145620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.146275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:02.147721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:02.172967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 519/1026] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:06:01 2022] POST /identity/v3/users => generated 285 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:06:02.189187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd09edd-352b-4ec6-8677-e746a4d496fd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:02.231832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:02.232509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.232788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.233095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] PATH_INFO: `/v3/services/image` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.233757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:02.235104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:02.245441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:02.246518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:02.247387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 508/1027] 10.222.0.26 () {56 vars in 1105 bytes} [Sat Oct 1 04:06:02 2022] GET /identity/v3/services/image => generated 86 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:02.254614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:02.365332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a405ef4-eef6-48c7-9c45-bcf5870ef00d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:02.401885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:02.402632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.403003 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.403369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.404105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] RBAC: Adding query filter params (name=image) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:02.404514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:02.405890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:02.412865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 520/1028] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:02 2022] GET /identity/v3/services?name=image => generated 122 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:02.418856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:02.513923 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:02.537979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f51214-e613-48ef-8474-a5a0052486c9 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:02.574255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5cf765-b3eb-4369-a54f-3a5066f632cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:02.574872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5cf765-b3eb-4369-a54f-3a5066f632cb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.575152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5cf765-b3eb-4369-a54f-3a5066f632cb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.575425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5cf765-b3eb-4369-a54f-3a5066f632cb None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.576061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5cf765-b3eb-4369-a54f-3a5066f632cb None admin] RBAC: Adding query filter params (type=image) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:02.576365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5cf765-b3eb-4369-a54f-3a5066f632cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:02.577759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5cf765-b3eb-4369-a54f-3a5066f632cb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:02.583119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 509/1029] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:02 2022] GET /identity/v3/services?type=image => generated 350 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:02.588810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:02.607728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98376cf-e5e7-42f5-bb71-044bae65081e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.608248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98376cf-e5e7-42f5-bb71-044bae65081e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.608531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98376cf-e5e7-42f5-bb71-044bae65081e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.610042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 510/1030] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:06:02.615825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3e6f45-8680-4bc7-b7eb-fd95a0809289 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.616115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3e6f45-8680-4bc7-b7eb-fd95a0809289 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.616390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3e6f45-8680-4bc7-b7eb-fd95a0809289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.635139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cd3e6f45-8680-4bc7-b7eb-fd95a0809289 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:02.636338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cd3e6f45-8680-4bc7-b7eb-fd95a0809289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:02.642893 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:02.702458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58774eaf-297f-4be8-b1bd-0d569c9f8da6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:02.707195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 511/1031] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:02 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:02.742867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:02.743891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.744394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.744867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.745792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] RBAC: Adding query filter params (interface=public, service_id=672482c05a604718bd50dc9566fcf1ca, region_id=RegionOne) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:02.746307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:02.747874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:02.758154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 521/1032] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:06:02 2022] GET /identity/v3/endpoints?service_id=672482c05a604718bd50dc9566fcf1ca&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:02.759865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3e6f45-8680-4bc7-b7eb-fd95a0809289 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:02.838363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:02.864017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3e6f45-8680-4bc7-b7eb-fd95a0809289 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:02.898694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:02.899710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:02.900195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:02.900672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:02.901516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:02.903072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:02.911029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 512/1033] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:06:02 2022] GET /identity/v3/services => generated 1957 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:06:02.979735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:03.092180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad592ad2-91f1-4800-a823-db118b7578dc None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:03.106131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:03.129241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:03.129876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:03.130152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:03.130421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:03.141772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:03.147461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:03.148790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:03.149845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 522/1034] 10.222.0.26 () {56 vars in 1120 bytes} [Sat Oct 1 04:06:02 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:03.154961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:03.256898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbc2e04-efb8-46bc-b3ac-95120cfae2c4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:03.291223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:03.291824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:03.292101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:03.292371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:03.293051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:03.293354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:03.294703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:03.302856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 513/1035] 10.222.0.26 () {56 vars in 1115 bytes} [Sat Oct 1 04:06:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:03.308588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:03.412797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de274c25-fc21-42a4-b321-af0eae62a6b9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:03.449564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:03.450187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:03.450466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:03.450734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:03.461345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:03.462769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:03.477073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:03.478908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:06:03.481100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:03.481100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:03.481100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:03.481100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 523/1036] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:03.484289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:03.487234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:03.594806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd4c0e-573b-40d6-9e3e-8c6ccb91b07f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:03.628531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:03.629190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:03.629406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:03.629673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:03.630305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:03.630603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:03.631949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:03.644486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 514/1037] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:06:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:03.649947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:03.676977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:03.751959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42defc64-3c92-4d35-bf5b-25ef1fe7036a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:03.785431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:03.786054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:03.786335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:03.786606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:03.793932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:03.795353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:06:03.806793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:03.807794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:06:03.808770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:03.812957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 524/1038] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:06:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:03.813746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:03.882389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:03.923905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d56d06-7340-4b76-a8de-d95bbd87f567 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:03.958809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:03.959343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:03.959620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:03.959889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:03.960614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:03.960921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:03.962360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:03.985245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 515/1039] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:06:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:03.986457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:04.075082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:04.092848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489303c4-1707-4e8d-9259-b1b2db34b994 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:04.126417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a4e3eb-585a-4d25-87f1-58fa37bf889d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:04.127015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a4e3eb-585a-4d25-87f1-58fa37bf889d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.127289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a4e3eb-585a-4d25-87f1-58fa37bf889d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.127556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a4e3eb-585a-4d25-87f1-58fa37bf889d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.193879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a4e3eb-585a-4d25-87f1-58fa37bf889d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:04.195323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a4e3eb-585a-4d25-87f1-58fa37bf889d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:04.278220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:04.299819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 525/1040] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:03 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 01 04:06:04.301320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:04.406359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b923060-af78-4246-b7c7-e7e64e8e7468 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:04.427757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:04.442713 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:04.443343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.443620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.443890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.448871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752abbc-92b9-452f-8f9b-4efc26a1c9e3 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.449194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752abbc-92b9-452f-8f9b-4efc26a1c9e3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.449475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7752abbc-92b9-452f-8f9b-4efc26a1c9e3 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.450950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 526/1041] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:06:04.453884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:04.455314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:04.455933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddfffe-f93c-4e64-b312-e437c532a783 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.456220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddfffe-f93c-4e64-b312-e437c532a783 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.456529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ddfffe-f93c-4e64-b312-e437c532a783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:04.470303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:04.471271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:06:04.472862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:04.472862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:04.472862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:04.472862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 516/1042] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:04.477617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-93ddfffe-f93c-4e64-b312-e437c532a783 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:04.477777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:04.478783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-93ddfffe-f93c-4e64-b312-e437c532a783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:04.561000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 527/1043] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:04 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:04.587986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfac5f1-b355-4b02-9498-fbf40096ffaf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:04.621951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:04.622565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.622877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.623092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.623730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:04.624033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:04.625425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:04.638265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 517/1044] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:06:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:04.643870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddfffe-f93c-4e64-b312-e437c532a783 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:04.745765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ddfffe-f93c-4e64-b312-e437c532a783 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:04.759714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:04.781544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:04.782183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.782463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.782992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.790335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:04.791773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:06:04.802864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:04.803785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:06:04.804695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:04.809000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 528/1045] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:06:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:04.809950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:04.833889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:04.842701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:04.917747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64286ed4-aec5-43b7-a046-e6db5f5a6ee0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:04.938478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df18009-35be-4c5a-8213-c2d0f1c5d2d5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:04.957501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:04.958179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.958463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.958735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.959426 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:04.959731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:04.961186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:04.974049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 518/1046] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:06:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:04.979543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:04.980190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:04.980475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:04.980718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:04.980816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:04.992583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:04.997430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:04.998371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:04.999239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 529/1047] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:05.004408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.047447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:05.084806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:05.094298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc36d9a-c0ad-415c-9775-db1bcc0de302 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.126740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a043d785-a6d2-4ac9-b054-fcaccbe7dff7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.131683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823bdf7a-4510-4cf3-8c2c-8b291d893cf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:05.132362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823bdf7a-4510-4cf3-8c2c-8b291d893cf8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.132709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823bdf7a-4510-4cf3-8c2c-8b291d893cf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.133078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823bdf7a-4510-4cf3-8c2c-8b291d893cf8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.133871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823bdf7a-4510-4cf3-8c2c-8b291d893cf8 None admin] RBAC: Adding query filter params (role.id=16de5c1385014299a8e7cef6b0114795, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=901b9bb47c2f40c2b1b4cfe7886a1167) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:05.134244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823bdf7a-4510-4cf3-8c2c-8b291d893cf8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:05.135787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823bdf7a-4510-4cf3-8c2c-8b291d893cf8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:05.152957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 519/1048] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:04 2022] GET /identity/v3/role_assignments?user.id=901b9bb47c2f40c2b1b4cfe7886a1167&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=16de5c1385014299a8e7cef6b0114795 => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:05.155008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c6d81-d9ff-45d5-b207-1eecf175dbf8 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.155293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c6d81-d9ff-45d5-b207-1eecf175dbf8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.155565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c6d81-d9ff-45d5-b207-1eecf175dbf8 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.157444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 520/1049] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:06:05.164084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15aefc0-6d1d-48c9-becf-09d5939a9fa6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.164342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15aefc0-6d1d-48c9-becf-09d5939a9fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.164620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15aefc0-6d1d-48c9-becf-09d5939a9fa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.170362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:05.171027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.171301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.171568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.172247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:05.172544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:05.173956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:05.184071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d15aefc0-6d1d-48c9-becf-09d5939a9fa6 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:05.185308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 530/1050] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:05.189409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d15aefc0-6d1d-48c9-becf-09d5939a9fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:05.191450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.276982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 521/1051] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:05 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:05.302305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:05.309012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c89312-bd9b-4b05-8f42-7c4885490b13 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.347978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:05.348676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.349094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.349479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] PATH_INFO: `/v3/users/a7b6711452dc4e14bf650c430a54fe98` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.363607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] RBAC: Authorizing `identity:get_user(user_id=a7b6711452dc4e14bf650c430a54fe98)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:05.365199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:05.371566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 531/1052] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/users/a7b6711452dc4e14bf650c430a54fe98 => generated 285 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:05.373089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15aefc0-6d1d-48c9-becf-09d5939a9fa6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.461428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:05.477278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15aefc0-6d1d-48c9-becf-09d5939a9fa6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.517706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:05.518458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.518833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.519211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.558703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.586851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:05.588306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:05.674899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e56e41-395f-4516-a898-6868cb80cec8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.675627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:05.693180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 522/1053] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:05.699282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.718089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:05.718726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.719002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.719270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] PATH_INFO: `/v3/services/image` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.719852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:05.723701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:05.732856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:05.733938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:05.735274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 532/1054] 10.222.0.26 () {56 vars in 1105 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/services/image => generated 86 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:05.740273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.763471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:05.813800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979d3324-5833-44d4-95a5-0f9f324a107f None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.848315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bde44ea-6b17-4a45-ba11-c367f780fa35 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.849345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:05.849987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.850291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.851313 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.851683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] RBAC: Authorizing `identity:get_project(project_id=02a46045c759418ab6500876b97ab36b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:05.853123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:05.854895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 523/1055] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/projects/02a46045c759418ab6500876b97ab36b => generated 311 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:05.860741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.889252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:05.889859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:05.890293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:05.890566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:05.891510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] RBAC: Adding query filter params (name=image) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:05.892074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:05.893982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:05.900529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 533/1056] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:05.905885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:05.920468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:05.975454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5cbea9-a7ce-4800-97ba-f8d83e42eb67 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:05.994022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:06.010772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:06.011617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:06.011747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:06.011926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:06.012652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=02a46045c759418ab6500876b97ab36b, user.id=a7b6711452dc4e14bf650c430a54fe98) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:06.012982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:06.014412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:06.015021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa960d0-48dd-4fdc-93ed-8a886579f004 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:06.021465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 524/1057] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/role_assignments?user.id=a7b6711452dc4e14bf650c430a54fe98&scope.project.id=02a46045c759418ab6500876b97ab36b&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:06.060397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b042401-8343-4cc9-8374-389a9fb8d5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:06.061153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b042401-8343-4cc9-8374-389a9fb8d5c7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:06.061503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b042401-8343-4cc9-8374-389a9fb8d5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:06.061847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b042401-8343-4cc9-8374-389a9fb8d5c7 None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:06.062602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b042401-8343-4cc9-8374-389a9fb8d5c7 None admin] RBAC: Adding query filter params (type=image) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:06.062977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b042401-8343-4cc9-8374-389a9fb8d5c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:06.064432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b042401-8343-4cc9-8374-389a9fb8d5c7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:06.072000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 534/1058] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:05 2022] GET /identity/v3/services?type=image => generated 350 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:06.077841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:06.190055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2ca624-1927-42de-8cb5-c811fd6d4764 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:06.210283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:06.227560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fe4dad-f213-4ed2-a55e-bc0368a78cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:06.228184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe4dad-f213-4ed2-a55e-bc0368a78cb2 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:06.228531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe4dad-f213-4ed2-a55e-bc0368a78cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:06.228869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe4dad-f213-4ed2-a55e-bc0368a78cb2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:06.229624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fe4dad-f213-4ed2-a55e-bc0368a78cb2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:06.231039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fe4dad-f213-4ed2-a55e-bc0368a78cb2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:06.261048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 525/1059] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:06:06 2022] POST /identity/v3/endpoints => generated 327 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:06:07.340599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42321da1-5c59-42dc-a482-3219c18742c1 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:07.341148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42321da1-5c59-42dc-a482-3219c18742c1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:07.341227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42321da1-5c59-42dc-a482-3219c18742c1 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:07.342699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 535/1060] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:07.347632 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c95d3b4-d66a-4948-ab24-27c57c76570a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:07.348020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c95d3b4-d66a-4948-ab24-27c57c76570a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:07.348194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c95d3b4-d66a-4948-ab24-27c57c76570a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:07.368247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8c95d3b4-d66a-4948-ab24-27c57c76570a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:07.369516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8c95d3b4-d66a-4948-ab24-27c57c76570a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:07.441315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 526/1061] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:07.711059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42321da1-5c59-42dc-a482-3219c18742c1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:07.813457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42321da1-5c59-42dc-a482-3219c18742c1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:07.848074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:07.848698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:07.848996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:07.849267 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:07.859185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:07.863923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:07.864864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:06:07.866535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:07.869288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 536/1062] 10.222.0.26 () {56 vars in 1120 bytes} [Sat Oct 1 04:06:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:07.871006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c95d3b4-d66a-4948-ab24-27c57c76570a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:07.966844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:07.970513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c95d3b4-d66a-4948-ab24-27c57c76570a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.006231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d28a5c-b05e-4628-81dd-b75a2da07840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.006847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d28a5c-b05e-4628-81dd-b75a2da07840 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.007125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d28a5c-b05e-4628-81dd-b75a2da07840 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.007397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d28a5c-b05e-4628-81dd-b75a2da07840 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.008057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d28a5c-b05e-4628-81dd-b75a2da07840 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:08.008360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d28a5c-b05e-4628-81dd-b75a2da07840 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:08.009768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d28a5c-b05e-4628-81dd-b75a2da07840 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:08.023061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 527/1063] 10.222.0.26 () {56 vars in 1115 bytes} [Sat Oct 1 04:06:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:08.024631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.034381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:08.131146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb71f9b-f0b8-49b9-b842-def5b579b576 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.165891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.166526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.166804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.167076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.177143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:08.178571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:08.193402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:08.194371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:06:08.200702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:08.200702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:08.200702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:08.200702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 537/1064] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:08.202168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b84224-92bb-49cd-98fd-86f267ff7ffd None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.202279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.202458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b84224-92bb-49cd-98fd-86f267ff7ffd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.202733 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b84224-92bb-49cd-98fd-86f267ff7ffd None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.204188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 528/1065] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:06:08.209625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0ac6f-d594-48e6-bee2-38c2a74a9daa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.209914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0ac6f-d594-48e6-bee2-38c2a74a9daa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.210193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0ac6f-d594-48e6-bee2-38c2a74a9daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.228125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d6f0ac6f-d594-48e6-bee2-38c2a74a9daa None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:08.229343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d6f0ac6f-d594-48e6-bee2-38c2a74a9daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:08.263936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:08.303343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 529/1066] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:08.315125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faaf0f6-96aa-4e69-8a66-3357b4830a99 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.349489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363e8b96-2c16-423f-8da9-42e527695153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.350165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363e8b96-2c16-423f-8da9-42e527695153 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.350405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363e8b96-2c16-423f-8da9-42e527695153 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.350632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363e8b96-2c16-423f-8da9-42e527695153 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.351269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363e8b96-2c16-423f-8da9-42e527695153 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:08.351571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363e8b96-2c16-423f-8da9-42e527695153 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:08.352944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363e8b96-2c16-423f-8da9-42e527695153 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:08.369982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 538/1067] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:08.371709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0ac6f-d594-48e6-bee2-38c2a74a9daa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.412367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79d52bf-9f52-4fbb-986e-08847c692569 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.412699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79d52bf-9f52-4fbb-986e-08847c692569 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.413886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79d52bf-9f52-4fbb-986e-08847c692569 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.415431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 539/1068] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:08.420630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf76bb-011c-4cf7-a18f-5af5ad89ae1a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.420908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf76bb-011c-4cf7-a18f-5af5ad89ae1a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.421221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bf76bb-011c-4cf7-a18f-5af5ad89ae1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.440029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f5bf76bb-011c-4cf7-a18f-5af5ad89ae1a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:08.441972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f5bf76bb-011c-4cf7-a18f-5af5ad89ae1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:08.474049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:08.483018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0ac6f-d594-48e6-bee2-38c2a74a9daa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.513065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 540/1069] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:08.529271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.530073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.530356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.530652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.538285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:08.539713 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:06:08.552332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:08.558176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:08.559390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 530/1070] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:08.561518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bf76bb-011c-4cf7-a18f-5af5ad89ae1a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.578851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.624045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:08.683483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bf76bb-011c-4cf7-a18f-5af5ad89ae1a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.695747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7560ab01-fa79-4ee7-bc6e-16b0160280ea None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.722846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.723542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.723730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.724005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.724845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:08.725190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:08.726611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:08.733129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.733709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.733991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.734265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.745606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 541/1071] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:08.747532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:08.751261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:08.752774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:08.754480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:08.755436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:08.760672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 531/1072] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:08.762133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.781041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:08.851074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03485b4a-8507-4b3e-99dc-627a31f124ef None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.858798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:08.867284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176913ae-b229-4360-addc-5c65821f2a72 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:08.885922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.886546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.886825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.887094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.896881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:08.898321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:08.900290 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:08.900890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:08.901189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:08.901458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:08.902107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:08.902406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:08.903774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:08.912877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 532/1073] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:08.913356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:08.914418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:06:08.915735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:08.915735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:08.915735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:08.915962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.916047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 542/1074] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:08.919027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:08.987983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.012631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f81b673-8209-429d-ab6a-2d38341ffae5 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.018096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fef76e7-ee30-4f11-9e54-0e2d9f456339 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.049408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.050638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.050919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.051185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.052333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.056911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.057239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.057522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] PATH_INFO: `/v3/users/a7b6711452dc4e14bf650c430a54fe98` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.058719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] RBAC: Authorizing `identity:get_user(user_id=a7b6711452dc4e14bf650c430a54fe98)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.060815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.062536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 543/1075] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/users/a7b6711452dc4e14bf650c430a54fe98 => generated 285 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:09.063308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.092602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.125875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.127326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.163658 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c83dc3-fb11-4cd5-8422-18a4c40672fd None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.195748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.196346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.196626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.196899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.197560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:09.197859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.199210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.201182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.212063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 544/1076] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:06:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:09.212857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.232870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 533/1077] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:09.233728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.296623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.311613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003619c0-f94d-4741-a7f6-ab05d4f7d71a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.332101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff84062-91ee-44fa-9321-9c55452a551d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.345867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.346488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.346806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.347757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.349021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] RBAC: Authorizing `identity:get_project(project_id=02a46045c759418ab6500876b97ab36b)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.350430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.352047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 545/1078] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:09 2022] GET /identity/v3/projects/02a46045c759418ab6500876b97ab36b => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:09.352810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.366176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.366799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.367073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.367343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.373883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.375305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:06:09.386159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:09.387230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 534/1079] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:06:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:09.388138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.424077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.452996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d055ba8-2473-4b6c-a738-257f87244d02 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.487287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.487952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.488147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc11b342-f01c-45cd-8f11-e45ef6ab8643 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.488261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.488484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.498166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.499594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:09.516675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:09.517725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:06:09.519205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:09.519205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:09.519205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:09.519205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 546/1080] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:09.519205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.528619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.531693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.533004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.533297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.533732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b/users/a7b6711452dc4e14bf650c430a54fe98/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.535645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] RBAC: Authorizing `identity:create_grant(project_id=02a46045c759418ab6500876b97ab36b, user_id=a7b6711452dc4e14bf650c430a54fe98, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.537124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.550436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 535/1081] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:06:09 2022] PUT /identity/v3/projects/02a46045c759418ab6500876b97ab36b/users/a7b6711452dc4e14bf650c430a54fe98/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:06:09.551217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.638058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174a4ed6-fa74-454d-9189-58f08dc01337 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.678907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.679591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.680227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.680501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.680767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.681523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:09.681831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.683353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.688809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fff1d7-c78a-4cd3-be0b-a084828d3d81 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.697532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 547/1082] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:06:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:09.703513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:09.734280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc55885-f8d2-4a1a-a8ec-55aa56801963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.735275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc55885-f8d2-4a1a-a8ec-55aa56801963 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.735784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc55885-f8d2-4a1a-a8ec-55aa56801963 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.736275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc55885-f8d2-4a1a-a8ec-55aa56801963 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.737170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc55885-f8d2-4a1a-a8ec-55aa56801963 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:09.737899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc55885-f8d2-4a1a-a8ec-55aa56801963 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.739360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc55885-f8d2-4a1a-a8ec-55aa56801963 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.751763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 536/1083] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:06:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:09.777701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:09.818069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c7e5d-3c4a-4350-9cd6-9b0584c9bb65 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:09.856142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b4c4d0-461e-41a3-a2bd-5d0bfed079a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:09.856812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4c4d0-461e-41a3-a2bd-5d0bfed079a2 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:09.857117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4c4d0-461e-41a3-a2bd-5d0bfed079a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:09.857389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b4c4d0-461e-41a3-a2bd-5d0bfed079a2 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/901b9bb47c2f40c2b1b4cfe7886a1167/roles/16de5c1385014299a8e7cef6b0114795` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:09.859575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4c4d0-461e-41a3-a2bd-5d0bfed079a2 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=901b9bb47c2f40c2b1b4cfe7886a1167, role_id=16de5c1385014299a8e7cef6b0114795)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:09.861037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b4c4d0-461e-41a3-a2bd-5d0bfed079a2 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:09.875981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 548/1084] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:06:09 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/901b9bb47c2f40c2b1b4cfe7886a1167/roles/16de5c1385014299a8e7cef6b0114795 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:06:11.753415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f080f071-fdca-4e8f-837f-887b54677fbf None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:11.753994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f080f071-fdca-4e8f-837f-887b54677fbf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:11.753994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f080f071-fdca-4e8f-837f-887b54677fbf None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:11.755298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 537/1085] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:11.760681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4cea5-624c-42c0-a50d-218ddedf8e70 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:11.761125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4cea5-624c-42c0-a50d-218ddedf8e70 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:11.761268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d4cea5-624c-42c0-a50d-218ddedf8e70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:11.779290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-38d4cea5-624c-42c0-a50d-218ddedf8e70 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:11.780398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-38d4cea5-624c-42c0-a50d-218ddedf8e70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:11.845695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 549/1086] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:12.042073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc0a35d-d484-44af-87a2-9c3ce20178e7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.042301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc0a35d-d484-44af-87a2-9c3ce20178e7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.042580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc0a35d-d484-44af-87a2-9c3ce20178e7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.043953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 538/1087] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:12.049603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd775df-b751-47fd-b734-ff29ffb6b220 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.050578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd775df-b751-47fd-b734-ff29ffb6b220 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.050965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd775df-b751-47fd-b734-ff29ffb6b220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.069199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8dd775df-b751-47fd-b734-ff29ffb6b220 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:12.070497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8dd775df-b751-47fd-b734-ff29ffb6b220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:12.124268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc0a35d-d484-44af-87a2-9c3ce20178e7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.124497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:12.141508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 550/1088] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:12.224215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc0a35d-d484-44af-87a2-9c3ce20178e7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.256118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:12.256738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.257047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.257318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.266907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:12.272135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:12.273323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:12.274267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 539/1089] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:12.279345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd775df-b751-47fd-b734-ff29ffb6b220 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.376794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd775df-b751-47fd-b734-ff29ffb6b220 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.409272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.410199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:12.410805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.411086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.411357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.412023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:12.412323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:12.413704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:12.421937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 551/1090] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:12.427814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:12.428109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.476651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:12.511593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b15405-1172-4072-bbc7-ce7d5667e8fd None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.530521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7890cec-9418-4c70-85fe-60d2284b3575 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.546353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:12.546978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.547284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.547557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.557742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:12.562406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:12.563368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:12.564376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 540/1091] 10.222.0.26 () {56 vars in 1120 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:12.565400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:12.565400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.565400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.565400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] PATH_INFO: `/v3/users/a7b6711452dc4e14bf650c430a54fe98` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.566466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] RBAC: Authorizing `identity:get_user(user_id=a7b6711452dc4e14bf650c430a54fe98)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:12.567891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:12.574310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 552/1092] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/users/a7b6711452dc4e14bf650c430a54fe98 => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:12.574454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.575078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.642677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:12.667545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf304cd-9c1d-4067-9971-3cb0767b0335 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.674019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886edd99-ab0c-48ab-8e88-3ce75777d9e6 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.689861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:12.700979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:12.701590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.701866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.702140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.702795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:12.703101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:12.704463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:12.710128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:12.710731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.711009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.711279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.712695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 541/1093] 10.222.0.26 () {56 vars in 1115 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:12.718333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.779139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:12.780616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:12.818453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba225eee-c386-4f92-96bf-1f76933e0a5a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.851272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:12.852294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:12.852578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:12.852850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:12.855817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:12.865128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:12.866675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:12.877534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 553/1094] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:12.880389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:12.881568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:06:12.882489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:12.882489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:12.882489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:12.882489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 542/1095] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:12.885885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.886867 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:12.903013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:12.984063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f2a7aa-40bc-409c-98b7-7460b85accd4 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:12.990229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a275c329-3484-49d5-97f9-29f0edd2ee50 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:13.019445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:13.020134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:13.020417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:13.020694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:13.021840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] RBAC: Authorizing `identity:get_project(project_id=02a46045c759418ab6500876b97ab36b)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:13.022347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:13.022949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:13.023228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:13.023321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:13.023500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:13.024132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:13.024435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:13.024870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 554/1096] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/projects/02a46045c759418ab6500876b97ab36b => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:13.025811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:13.030948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:13.041268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 543/1097] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:06:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:13.046720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:13.094808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:13.115326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:13.132755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f46d50-71e0-433d-b86c-4c75f0c1bdef None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:13.147632 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03658ec8-e86a-4d93-ba6b-8abcd8e1293b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:13.167323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:13.167939 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:13.168226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:13.168497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:13.169251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=02a46045c759418ab6500876b97ab36b, user.id=a7b6711452dc4e14bf650c430a54fe98) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:13.169562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:13.170970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:13.176772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 555/1098] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:13 2022] GET /identity/v3/role_assignments?user.id=a7b6711452dc4e14bf650c430a54fe98&scope.project.id=02a46045c759418ab6500876b97ab36b&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:13.181805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:13.182455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:13.182738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:13.183014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:13.191268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:13.192754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:06:13.206985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:13.208251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:13.209262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 544/1099] 10.222.0.26 () {56 vars in 1093 bytes} [Sat Oct 1 04:06:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:13.214548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:13.320597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b33a7-c421-46e0-b1a8-771c44ff00db None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:13.359577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:13.360187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:13.360467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:13.360748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:13.361593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:13.361900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:13.367606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:13.388451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 556/1100] 10.222.0.26 () {56 vars in 1133 bytes} [Sat Oct 1 04:06:13 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:06:13.395550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:13.496555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1fe099-057c-41f4-a72f-9cec77ef6e22 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:13.507139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:13.533050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:13.533667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:13.533951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:13.534384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:13.599863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:13.601344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:13.696161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 545/1101] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 01 04:06:13.702042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:13.711319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:13.802695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b44384d-5e96-4459-90bd-ee66d3d6043d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:13.835978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:13.836595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:13.836875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:13.837174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:13.847524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:13.848984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:13.863134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:13.864212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:06:13.865588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:13.865588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:06:13.865588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:13.865588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 557/1102] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:06:13.869933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:13.880123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:13.968089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7017abb-7fab-43dd-9334-17a7328e0e28 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:14.000803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:14.001431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:14.001710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:14.001982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:14.002619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:14.003143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:14.004772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:14.015899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 546/1103] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:06:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:14.021703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:14.078287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:14.122391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48436bc1-983d-4cba-9514-fec4acd103d9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:14.154014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:14.154627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:14.154900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:14.155167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:14.162246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:14.163653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:06:14.174367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:14.176094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:14.177050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 558/1104] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:06:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:14.181907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:14.246187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:14.278901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b0e822-7417-4cea-a08a-8440b7d92936 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:14.312180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280e5db2-d461-41b3-8886-730e50551704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:14.312788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280e5db2-d461-41b3-8886-730e50551704 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:14.313102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280e5db2-d461-41b3-8886-730e50551704 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:14.313376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280e5db2-d461-41b3-8886-730e50551704 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:14.314060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280e5db2-d461-41b3-8886-730e50551704 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:14.315160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280e5db2-d461-41b3-8886-730e50551704 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:14.316643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280e5db2-d461-41b3-8886-730e50551704 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:14.328468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 547/1105] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:06:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:14.334266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:14.434838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff860d7-d0d7-4be5-ade2-7449eb836387 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:14.452026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:14.469903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010388b6-f95f-4e59-a45b-de1eba48ca54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:14.470526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010388b6-f95f-4e59-a45b-de1eba48ca54 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:14.470800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010388b6-f95f-4e59-a45b-de1eba48ca54 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:14.471066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010388b6-f95f-4e59-a45b-de1eba48ca54 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:14.471926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010388b6-f95f-4e59-a45b-de1eba48ca54 None admin] RBAC: Adding query filter params (role.id=16de5c1385014299a8e7cef6b0114795, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=901b9bb47c2f40c2b1b4cfe7886a1167) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:14.472241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010388b6-f95f-4e59-a45b-de1eba48ca54 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:14.473669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010388b6-f95f-4e59-a45b-de1eba48ca54 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:14.479472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 559/1106] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:14 2022] GET /identity/v3/role_assignments?user.id=901b9bb47c2f40c2b1b4cfe7886a1167&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=16de5c1385014299a8e7cef6b0114795 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:15.380635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aa581b-47f3-474b-9979-66d3691b0f45 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:15.381188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aa581b-47f3-474b-9979-66d3691b0f45 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:15.381188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aa581b-47f3-474b-9979-66d3691b0f45 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:15.382530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 548/1107] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:15.387704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66303100-eae9-4a30-ab78-7fa2ca13e626 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:15.388120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66303100-eae9-4a30-ab78-7fa2ca13e626 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:15.388249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66303100-eae9-4a30-ab78-7fa2ca13e626 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:15.406148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-66303100-eae9-4a30-ab78-7fa2ca13e626 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:15.408147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-66303100-eae9-4a30-ab78-7fa2ca13e626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:15.477889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 560/1108] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:15.747475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aa581b-47f3-474b-9979-66d3691b0f45 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:15.845918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aa581b-47f3-474b-9979-66d3691b0f45 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:15.872577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:15.880553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a43ea8-2b15-4731-8808-ae066d586aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:15.881216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a43ea8-2b15-4731-8808-ae066d586aa7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:15.881498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a43ea8-2b15-4731-8808-ae066d586aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:15.881771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a43ea8-2b15-4731-8808-ae066d586aa7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:15.882868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a43ea8-2b15-4731-8808-ae066d586aa7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:15.884286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a43ea8-2b15-4731-8808-ae066d586aa7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:15.885846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 549/1109] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:15.891551 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66303100-eae9-4a30-ab78-7fa2ca13e626 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:15.985633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66303100-eae9-4a30-ab78-7fa2ca13e626 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:15.998063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:16.019230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3088f-355b-4818-a462-b3aba5bf374c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:16.019819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3088f-355b-4818-a462-b3aba5bf374c None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:16.020094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3088f-355b-4818-a462-b3aba5bf374c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:16.020362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3088f-355b-4818-a462-b3aba5bf374c None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:16.021034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3088f-355b-4818-a462-b3aba5bf374c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:16.022440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d3088f-355b-4818-a462-b3aba5bf374c None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:16.044074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 561/1110] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:06:15 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:06:18.206245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad2470-5735-4eb5-beb6-b10645b2da17 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:18.206829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad2470-5735-4eb5-beb6-b10645b2da17 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:18.206829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad2470-5735-4eb5-beb6-b10645b2da17 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:18.208111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 550/1111] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:18.213341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eaaba7-d736-4ce3-85a4-81732bc1b65b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:18.213778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eaaba7-d736-4ce3-85a4-81732bc1b65b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:18.213878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eaaba7-d736-4ce3-85a4-81732bc1b65b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:18.231743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a8eaaba7-d736-4ce3-85a4-81732bc1b65b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:18.232907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a8eaaba7-d736-4ce3-85a4-81732bc1b65b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:18.296002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 562/1112] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:18.567488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ad2470-5735-4eb5-beb6-b10645b2da17 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:18.664382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ad2470-5735-4eb5-beb6-b10645b2da17 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:18.696475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:18.697119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:18.697398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:18.697664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] PATH_INFO: `/v3/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:18.699000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] RBAC: Authorizing `identity:get_role(role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:18.700338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:18.701648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 551/1113] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:18 2022] GET /identity/v3/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:18.708245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eaaba7-d736-4ce3-85a4-81732bc1b65b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:18.799999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:18.804579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eaaba7-d736-4ce3-85a4-81732bc1b65b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:18.842435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:18.843058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:18.843335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:18.843602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] PATH_INFO: `/v3/users/740c795b98144edc9818bc220ba9de86` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:18.856643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] RBAC: Authorizing `identity:get_user(user_id=740c795b98144edc9818bc220ba9de86)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:18.858174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:18.859756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 563/1114] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:18 2022] GET /identity/v3/users/740c795b98144edc9818bc220ba9de86 => generated 286 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:18.865825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:18.925793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:18.961480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b35d9a-e5a3-4719-a2a6-678637613895 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:18.993775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:18.994370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:18.994645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:18.994914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:19.067470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:19.068839 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:19.126066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:19.164715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 552/1115] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:19.170775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:19.265152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90115c7d-3c88-468f-be90-e125afb7426d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:19.298254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9626c31-b6dd-4a4f-94d3-b3e17526dba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:19.298873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9626c31-b6dd-4a4f-94d3-b3e17526dba9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:19.299154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9626c31-b6dd-4a4f-94d3-b3e17526dba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:19.299976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9626c31-b6dd-4a4f-94d3-b3e17526dba9 None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:19.301113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9626c31-b6dd-4a4f-94d3-b3e17526dba9 None admin] RBAC: Authorizing `identity:get_project(project_id=02a46045c759418ab6500876b97ab36b)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:19.302525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9626c31-b6dd-4a4f-94d3-b3e17526dba9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:19.304114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 564/1116] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:19 2022] GET /identity/v3/projects/02a46045c759418ab6500876b97ab36b => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:19.310225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:19.408210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba1365-78d3-4b51-986f-57b5ccde2a29 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:19.443229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06082c91-20b9-452a-a9bb-eba1746d26fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:19.443855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06082c91-20b9-452a-a9bb-eba1746d26fb None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:19.444135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06082c91-20b9-452a-a9bb-eba1746d26fb None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:19.444407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06082c91-20b9-452a-a9bb-eba1746d26fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:19.445153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06082c91-20b9-452a-a9bb-eba1746d26fb None admin] RBAC: Adding query filter params (role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=02a46045c759418ab6500876b97ab36b, user.id=740c795b98144edc9818bc220ba9de86) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:19.445464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06082c91-20b9-452a-a9bb-eba1746d26fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:19.446885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06082c91-20b9-452a-a9bb-eba1746d26fb None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:19.452101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 553/1117] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:19 2022] GET /identity/v3/role_assignments?user.id=740c795b98144edc9818bc220ba9de86&scope.project.id=02a46045c759418ab6500876b97ab36b&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:21.587451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e6e0c9-c654-416e-bc43-a76f3cfd0176 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:21.587993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e6e0c9-c654-416e-bc43-a76f3cfd0176 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:21.587993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e6e0c9-c654-416e-bc43-a76f3cfd0176 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:21.589465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 565/1118] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:21.594455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3884cf2a-c0a3-4686-8ac8-2d4a6e81f5a0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:21.594856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3884cf2a-c0a3-4686-8ac8-2d4a6e81f5a0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:21.595005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3884cf2a-c0a3-4686-8ac8-2d4a6e81f5a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:21.613964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3884cf2a-c0a3-4686-8ac8-2d4a6e81f5a0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:21.615150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3884cf2a-c0a3-4686-8ac8-2d4a6e81f5a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:21.636770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:21.680346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 554/1119] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:21.949325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e6e0c9-c654-416e-bc43-a76f3cfd0176 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:21.974804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:22.048484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e6e0c9-c654-416e-bc43-a76f3cfd0176 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:22.080314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:22.081494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:22.081761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:22.082026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] PATH_INFO: `/v3/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:22.083394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] RBAC: Authorizing `identity:get_role(role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:22.084736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:22.086092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 566/1120] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:21 2022] GET /identity/v3/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:22.091887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3884cf2a-c0a3-4686-8ac8-2d4a6e81f5a0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:22.191242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3884cf2a-c0a3-4686-8ac8-2d4a6e81f5a0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:22.224275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:22.224833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:22.225142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:22.225417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] PATH_INFO: `/v3/users/740c795b98144edc9818bc220ba9de86` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:22.226509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] RBAC: Authorizing `identity:get_user(user_id=740c795b98144edc9818bc220ba9de86)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:22.227928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:22.229509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 555/1121] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:22 2022] GET /identity/v3/users/740c795b98144edc9818bc220ba9de86 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:22.235302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:22.323105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:22.331820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d7d59-d029-4267-9a79-1b5901f65fe5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:22.364430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc02c8e1-8e94-4ba6-84f1-355b5407e11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:22.365064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc02c8e1-8e94-4ba6-84f1-355b5407e11e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:22.365340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc02c8e1-8e94-4ba6-84f1-355b5407e11e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:22.365613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc02c8e1-8e94-4ba6-84f1-355b5407e11e None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:22.366694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc02c8e1-8e94-4ba6-84f1-355b5407e11e None admin] RBAC: Authorizing `identity:get_project(project_id=02a46045c759418ab6500876b97ab36b)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:22.368088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc02c8e1-8e94-4ba6-84f1-355b5407e11e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:22.369687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 567/1122] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:22 2022] GET /identity/v3/projects/02a46045c759418ab6500876b97ab36b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:22.375511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:22.402748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:22.472726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f549abf-b9f6-4da4-9c77-25e6af15971d None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:22.504453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962315ad-c887-4b83-a9da-983c6a9e002e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:22.505096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962315ad-c887-4b83-a9da-983c6a9e002e None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:22.505378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962315ad-c887-4b83-a9da-983c6a9e002e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:22.505647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962315ad-c887-4b83-a9da-983c6a9e002e None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b/users/740c795b98144edc9818bc220ba9de86/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:22.507732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962315ad-c887-4b83-a9da-983c6a9e002e None admin] RBAC: Authorizing `identity:create_grant(project_id=02a46045c759418ab6500876b97ab36b, user_id=740c795b98144edc9818bc220ba9de86, role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:22.509174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962315ad-c887-4b83-a9da-983c6a9e002e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:22.522518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 556/1123] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:06:22 2022] PUT /identity/v3/projects/02a46045c759418ab6500876b97ab36b/users/740c795b98144edc9818bc220ba9de86/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:06:24.618435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d55274-2828-4b2f-8dc2-8ac0a8695ccb None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:24.618983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d55274-2828-4b2f-8dc2-8ac0a8695ccb None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:24.618983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d55274-2828-4b2f-8dc2-8ac0a8695ccb None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:24.620299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 568/1124] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:24.625771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aa4a2-6489-465b-9aba-5b7fdf4e8426 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:24.626258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aa4a2-6489-465b-9aba-5b7fdf4e8426 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:24.626351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aa4a2-6489-465b-9aba-5b7fdf4e8426 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:24.644002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-310aa4a2-6489-465b-9aba-5b7fdf4e8426 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:24.645213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-310aa4a2-6489-465b-9aba-5b7fdf4e8426 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:24.709233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 557/1125] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:24.980258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d55274-2828-4b2f-8dc2-8ac0a8695ccb None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:25.077366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d55274-2828-4b2f-8dc2-8ac0a8695ccb None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:25.109866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:25.110482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:25.110761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:25.111035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] PATH_INFO: `/v3/roles/a7dd23b32f02499fb312d5ef193d1d72` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:25.112956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] RBAC: Authorizing `identity:get_role(role_id=a7dd23b32f02499fb312d5ef193d1d72)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:25.114304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:25.115585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 569/1126] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:24 2022] GET /identity/v3/roles/a7dd23b32f02499fb312d5ef193d1d72 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:25.121173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aa4a2-6489-465b-9aba-5b7fdf4e8426 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:25.138541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:25.217062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aa4a2-6489-465b-9aba-5b7fdf4e8426 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:25.249287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:25.249845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:25.250121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:25.250393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] PATH_INFO: `/v3/users/740c795b98144edc9818bc220ba9de86` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:25.251482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] RBAC: Authorizing `identity:get_user(user_id=740c795b98144edc9818bc220ba9de86)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:25.253645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:25.255166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 558/1127] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:25 2022] GET /identity/v3/users/740c795b98144edc9818bc220ba9de86 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:25.260841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:25.290461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:25.358547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd7239-1a6f-43fd-a966-0f05e532aae9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:25.391040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:25.391630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:25.391908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:25.392178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:25.462074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:25.463502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:25.489521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:25.559337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 570/1128] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:25 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:25.565347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:25.645098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:25.661581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc44ef8-a6e0-4e4b-a4af-60e20c11767c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:25.693399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92c2fcc-554b-417c-a843-1be537205d99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:25.694012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c2fcc-554b-417c-a843-1be537205d99 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:25.694286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c2fcc-554b-417c-a843-1be537205d99 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:25.694570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c2fcc-554b-417c-a843-1be537205d99 None admin] PATH_INFO: `/v3/projects/02a46045c759418ab6500876b97ab36b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:25.695731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92c2fcc-554b-417c-a843-1be537205d99 None admin] RBAC: Authorizing `identity:get_project(project_id=02a46045c759418ab6500876b97ab36b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:25.697174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92c2fcc-554b-417c-a843-1be537205d99 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:25.698743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 559/1129] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:25 2022] GET /identity/v3/projects/02a46045c759418ab6500876b97ab36b => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:25.704779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:25.816989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f3909-835a-499c-a34c-9c4ed6cda43d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:25.843168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:25.852328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0edf7d4-0adc-40c9-9183-75645ba176f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:25.852937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0edf7d4-0adc-40c9-9183-75645ba176f9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:25.853227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0edf7d4-0adc-40c9-9183-75645ba176f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:25.853497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0edf7d4-0adc-40c9-9183-75645ba176f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:25.854215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0edf7d4-0adc-40c9-9183-75645ba176f9 None admin] RBAC: Adding query filter params (role.id=a7dd23b32f02499fb312d5ef193d1d72, scope.project.id=02a46045c759418ab6500876b97ab36b, user.id=740c795b98144edc9818bc220ba9de86) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:25.854523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0edf7d4-0adc-40c9-9183-75645ba176f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:25.855922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0edf7d4-0adc-40c9-9183-75645ba176f9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:25.861451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 571/1130] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:25 2022] GET /identity/v3/role_assignments?user.id=740c795b98144edc9818bc220ba9de86&scope.project.id=02a46045c759418ab6500876b97ab36b&role.id=a7dd23b32f02499fb312d5ef193d1d72 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:28.036564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f47766e-ca07-4ea0-8d64-d7e3e0b662fa None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:28.037159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f47766e-ca07-4ea0-8d64-d7e3e0b662fa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:28.037159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f47766e-ca07-4ea0-8d64-d7e3e0b662fa None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:28.038440 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 560/1131] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:28.043693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e97e19-11e2-440e-9b6e-9883f237e4c3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:28.044088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e97e19-11e2-440e-9b6e-9883f237e4c3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:28.044240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e97e19-11e2-440e-9b6e-9883f237e4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:28.062430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b1e97e19-11e2-440e-9b6e-9883f237e4c3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:28.063601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b1e97e19-11e2-440e-9b6e-9883f237e4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:28.129446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 572/1132] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:28.400662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f47766e-ca07-4ea0-8d64-d7e3e0b662fa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:28.499291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f47766e-ca07-4ea0-8d64-d7e3e0b662fa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:28.532607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de3474e-d6e4-4d5d-87b9-80d023d1dce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:28.533815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de3474e-d6e4-4d5d-87b9-80d023d1dce9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:28.534169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de3474e-d6e4-4d5d-87b9-80d023d1dce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:28.534438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de3474e-d6e4-4d5d-87b9-80d023d1dce9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:28.535522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de3474e-d6e4-4d5d-87b9-80d023d1dce9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:28.536959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de3474e-d6e4-4d5d-87b9-80d023d1dce9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:28.538464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 561/1133] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:28.544255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e97e19-11e2-440e-9b6e-9883f237e4c3 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:28.641529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e97e19-11e2-440e-9b6e-9883f237e4c3 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:28.664190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:28.675389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd614ca6-a5a0-4f20-ba97-60fe39d522b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:28.675978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd614ca6-a5a0-4f20-ba97-60fe39d522b1 None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:28.676252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd614ca6-a5a0-4f20-ba97-60fe39d522b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:28.676523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd614ca6-a5a0-4f20-ba97-60fe39d522b1 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:28.677201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd614ca6-a5a0-4f20-ba97-60fe39d522b1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:28.678610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd614ca6-a5a0-4f20-ba97-60fe39d522b1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:28.704201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 573/1134] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:06:28 2022] POST /identity/v3/projects => generated 311 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:06:30.854507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f2f496-b099-4b2d-a44a-edaca45eb8ec None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:30.855025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f2f496-b099-4b2d-a44a-edaca45eb8ec None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:30.855025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f2f496-b099-4b2d-a44a-edaca45eb8ec None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:30.856271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 562/1135] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:30.862200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b8bbe6-9d84-413b-9516-12bdaa037d61 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:30.862599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b8bbe6-9d84-413b-9516-12bdaa037d61 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:30.862736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b8bbe6-9d84-413b-9516-12bdaa037d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:30.880438 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-06b8bbe6-9d84-413b-9516-12bdaa037d61 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:30.881640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-06b8bbe6-9d84-413b-9516-12bdaa037d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:30.946397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 574/1136] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:06:31.219059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f2f496-b099-4b2d-a44a-edaca45eb8ec None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:31.240679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:31.319335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f2f496-b099-4b2d-a44a-edaca45eb8ec None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:31.351912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837015df-1da0-4e72-a31a-816af1b6a506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:31.352533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837015df-1da0-4e72-a31a-816af1b6a506 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:31.352811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837015df-1da0-4e72-a31a-816af1b6a506 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:31.353110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837015df-1da0-4e72-a31a-816af1b6a506 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:31.354194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837015df-1da0-4e72-a31a-816af1b6a506 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:31.355605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837015df-1da0-4e72-a31a-816af1b6a506 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:31.357153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 563/1137] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:06:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:31.363410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b8bbe6-9d84-413b-9516-12bdaa037d61 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:31.458211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b8bbe6-9d84-413b-9516-12bdaa037d61 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:31.460391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:31.492548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a3b4a-c6a7-490b-b97f-78d592d7aeee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:31.493092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a3b4a-c6a7-490b-b97f-78d592d7aeee None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:31.493369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a3b4a-c6a7-490b-b97f-78d592d7aeee None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:31.493633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a3b4a-c6a7-490b-b97f-78d592d7aeee None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:31.494281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a3b4a-c6a7-490b-b97f-78d592d7aeee None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:31.495697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a3b4a-c6a7-490b-b97f-78d592d7aeee None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:31.518482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 575/1138] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:06:31 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:06:33.698474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370db13-7612-42c8-b0cc-821edfccab94 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:33.699031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370db13-7612-42c8-b0cc-821edfccab94 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:33.699031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370db13-7612-42c8-b0cc-821edfccab94 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:33.700362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 564/1139] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:33.706043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521447c5-ac7a-402a-9067-dd0b62028866 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:33.706462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521447c5-ac7a-402a-9067-dd0b62028866 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:33.706599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521447c5-ac7a-402a-9067-dd0b62028866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:33.725211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-521447c5-ac7a-402a-9067-dd0b62028866 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:33.726384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-521447c5-ac7a-402a-9067-dd0b62028866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:33.791600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 576/1140] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:34.079637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1370db13-7612-42c8-b0cc-821edfccab94 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:34.165651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:34.178108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1370db13-7612-42c8-b0cc-821edfccab94 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:34.211685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:34.212311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:34.212600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:34.212873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:34.223183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:34.227894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:34.228998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:34.230043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 565/1141] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:34.235179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521447c5-ac7a-402a-9067-dd0b62028866 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:34.321362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:34.331814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521447c5-ac7a-402a-9067-dd0b62028866 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:34.366191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:34.366793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:34.367072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:34.367345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:34.368004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:34.368307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:34.369692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:34.377597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 577/1142] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:34.383539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:34.481937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29b06df-c15d-499d-a1a9-19822a4d0fa0 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:34.514825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:34.515450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:34.515731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:34.516001 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] PATH_INFO: `/v3/users/d5115206a65945c7af922e91bfa9dcaf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:34.528893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] RBAC: Authorizing `identity:get_user(user_id=d5115206a65945c7af922e91bfa9dcaf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:34.530356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:34.531940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 566/1143] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:34 2022] GET /identity/v3/users/d5115206a65945c7af922e91bfa9dcaf => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:34.538499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:34.635124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54b6dca-67df-4590-86be-672eee6108ff None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:34.668645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:34.669257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:34.669534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:34.669802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:34.689737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:34.734297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:34.735740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:34.828051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 578/1144] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:34.834515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:34.843127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:34.933025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3edb77-e011-4bbb-b90e-aa97546c6492 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:34.965807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb987de-3b9a-4797-b99f-46ffa3f5c0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:34.966431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb987de-3b9a-4797-b99f-46ffa3f5c0c4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:34.966712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb987de-3b9a-4797-b99f-46ffa3f5c0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:34.966986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb987de-3b9a-4797-b99f-46ffa3f5c0c4 None admin] PATH_INFO: `/v3/projects/66c431a7c3ca427b8be80e91bc4dd624` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:34.968082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb987de-3b9a-4797-b99f-46ffa3f5c0c4 None admin] RBAC: Authorizing `identity:get_project(project_id=66c431a7c3ca427b8be80e91bc4dd624)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:34.969533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb987de-3b9a-4797-b99f-46ffa3f5c0c4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:34.972135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 567/1145] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:34 2022] GET /identity/v3/projects/66c431a7c3ca427b8be80e91bc4dd624 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:34.978163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:35.028491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:35.073732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd30da1-401e-48fe-a052-98db0b9987e8 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:35.106306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd8f830-c011-4956-8bb5-14c3d5b2cf2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:35.106899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd8f830-c011-4956-8bb5-14c3d5b2cf2a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:35.107178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd8f830-c011-4956-8bb5-14c3d5b2cf2a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:35.107448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd8f830-c011-4956-8bb5-14c3d5b2cf2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:35.108160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd8f830-c011-4956-8bb5-14c3d5b2cf2a None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=66c431a7c3ca427b8be80e91bc4dd624, user.id=d5115206a65945c7af922e91bfa9dcaf) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:35.108469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd8f830-c011-4956-8bb5-14c3d5b2cf2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:35.109905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd8f830-c011-4956-8bb5-14c3d5b2cf2a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:35.115151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 579/1146] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:34 2022] GET /identity/v3/role_assignments?user.id=d5115206a65945c7af922e91bfa9dcaf&scope.project.id=66c431a7c3ca427b8be80e91bc4dd624&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:37.230981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a078601-181c-4017-8c30-5134b63480f6 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:37.231513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a078601-181c-4017-8c30-5134b63480f6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:37.231513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a078601-181c-4017-8c30-5134b63480f6 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:37.232849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 568/1147] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:37.238694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e18edb-00e8-4b58-95ee-e49a79122bb4 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:37.239135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e18edb-00e8-4b58-95ee-e49a79122bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:37.239228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e18edb-00e8-4b58-95ee-e49a79122bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:37.257496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-22e18edb-00e8-4b58-95ee-e49a79122bb4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:37.258667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-22e18edb-00e8-4b58-95ee-e49a79122bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:37.322678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 580/1148] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:37.601537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a078601-181c-4017-8c30-5134b63480f6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:37.678019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:37.703198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a078601-181c-4017-8c30-5134b63480f6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:37.735529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:37.737240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:37.737529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:37.737804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:37.749382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:37.754845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:37.755930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:37.757134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 569/1149] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:37.761938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e18edb-00e8-4b58-95ee-e49a79122bb4 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:37.807062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:37.858307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e18edb-00e8-4b58-95ee-e49a79122bb4 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:37.891336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:37.891930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:37.892208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:37.892502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:37.893181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:37.893483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:37.894827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:37.902460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 581/1150] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:37.908533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:38.005476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edef4a4-cc2a-402f-b2a7-4d083d6c77e4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:38.039250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:38.039865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:38.040144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:38.040418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] PATH_INFO: `/v3/users/d5115206a65945c7af922e91bfa9dcaf` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:38.041525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] RBAC: Authorizing `identity:get_user(user_id=d5115206a65945c7af922e91bfa9dcaf)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:38.042939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:38.044465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 570/1151] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:37 2022] GET /identity/v3/users/d5115206a65945c7af922e91bfa9dcaf => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:38.050170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:38.148489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97a02cc-3bd6-4a6e-88a2-258a5af044e0 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:38.156765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:38.181678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ef46d0-996c-45af-8231-1acc46d6eb87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:38.182300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ef46d0-996c-45af-8231-1acc46d6eb87 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:38.182580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ef46d0-996c-45af-8231-1acc46d6eb87 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:38.182850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ef46d0-996c-45af-8231-1acc46d6eb87 None admin] PATH_INFO: `/v3/projects/66c431a7c3ca427b8be80e91bc4dd624` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:38.183938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ef46d0-996c-45af-8231-1acc46d6eb87 None admin] RBAC: Authorizing `identity:get_project(project_id=66c431a7c3ca427b8be80e91bc4dd624)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:38.185378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ef46d0-996c-45af-8231-1acc46d6eb87 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:38.186978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 582/1152] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:38 2022] GET /identity/v3/projects/66c431a7c3ca427b8be80e91bc4dd624 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:38.193534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:38.196334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:38.289390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e011ed62-c325-4d4d-b781-2bd8d0b245a4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:38.321015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e654c87c-f57a-4fe4-abee-b4cba0341004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:38.321653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e654c87c-f57a-4fe4-abee-b4cba0341004 None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:38.321929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e654c87c-f57a-4fe4-abee-b4cba0341004 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:38.322198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e654c87c-f57a-4fe4-abee-b4cba0341004 None admin] PATH_INFO: `/v3/projects/66c431a7c3ca427b8be80e91bc4dd624/users/d5115206a65945c7af922e91bfa9dcaf/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:38.324033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e654c87c-f57a-4fe4-abee-b4cba0341004 None admin] RBAC: Authorizing `identity:create_grant(project_id=66c431a7c3ca427b8be80e91bc4dd624, user_id=d5115206a65945c7af922e91bfa9dcaf, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:38.325472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e654c87c-f57a-4fe4-abee-b4cba0341004 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:38.338606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 571/1153] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:06:38 2022] PUT /identity/v3/projects/66c431a7c3ca427b8be80e91bc4dd624/users/d5115206a65945c7af922e91bfa9dcaf/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:06:40.493579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07316e-bbc4-4417-a908-eb4745fa6bf5 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:40.494113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07316e-bbc4-4417-a908-eb4745fa6bf5 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:40.494113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b07316e-bbc4-4417-a908-eb4745fa6bf5 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:40.495506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 583/1154] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:40.500805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbbe49b-f139-4967-8ae5-7654c3f44ae8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:40.501201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbbe49b-f139-4967-8ae5-7654c3f44ae8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:40.501408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbbe49b-f139-4967-8ae5-7654c3f44ae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:40.520881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7bbbe49b-f139-4967-8ae5-7654c3f44ae8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:40.521560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7bbbe49b-f139-4967-8ae5-7654c3f44ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:40.576130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:40.587979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 572/1155] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:40.860822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07316e-bbc4-4417-a908-eb4745fa6bf5 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:40.959946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b07316e-bbc4-4417-a908-eb4745fa6bf5 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:40.992476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:40.993128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:40.993412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:40.993687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:41.003940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:41.008646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:41.009820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:41.010696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 584/1156] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:41.015747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbbe49b-f139-4967-8ae5-7654c3f44ae8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:41.111704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbbe49b-f139-4967-8ae5-7654c3f44ae8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:41.144142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:41.144741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:41.145044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:41.145314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:41.145968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:41.146269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:41.147609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:41.155603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 573/1157] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:41.162866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:41.197559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:41.264797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dafd3a-536a-4bd1-860f-48420d661510 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:41.299495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:41.300117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:41.300395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:41.300666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] PATH_INFO: `/v3/users/d5115206a65945c7af922e91bfa9dcaf` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:41.301791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] RBAC: Authorizing `identity:get_user(user_id=d5115206a65945c7af922e91bfa9dcaf)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:41.303212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:41.304766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 585/1158] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:41 2022] GET /identity/v3/users/d5115206a65945c7af922e91bfa9dcaf => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:41.310739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:41.364665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:41.408913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee82b74f-19ba-4bd0-88b2-69c32c39b537 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:41.441907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:41.442500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:41.442781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:41.443054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:41.507453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:41.508895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:41.564404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:41.607555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 574/1159] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:41.614287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:41.709788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d354d51-1d9b-4c29-9271-98e57fdfa11b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:41.712002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:41.742725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff09100b-0c2e-463d-a8dc-2be3b4ffb2fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:41.743344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff09100b-0c2e-463d-a8dc-2be3b4ffb2fc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:41.743624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff09100b-0c2e-463d-a8dc-2be3b4ffb2fc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:41.743898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff09100b-0c2e-463d-a8dc-2be3b4ffb2fc None admin] PATH_INFO: `/v3/projects/66c431a7c3ca427b8be80e91bc4dd624` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:41.745041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff09100b-0c2e-463d-a8dc-2be3b4ffb2fc None admin] RBAC: Authorizing `identity:get_project(project_id=66c431a7c3ca427b8be80e91bc4dd624)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:41.746463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff09100b-0c2e-463d-a8dc-2be3b4ffb2fc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:41.748053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 586/1160] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:41 2022] GET /identity/v3/projects/66c431a7c3ca427b8be80e91bc4dd624 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:41.754860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:41.850021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1a4357-06d3-4ff3-aa2b-6af0a4b368a2 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:41.882684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967c5fd6-5ffe-40e6-b322-5d6867b38297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:41.883286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c5fd6-5ffe-40e6-b322-5d6867b38297 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:41.883565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c5fd6-5ffe-40e6-b322-5d6867b38297 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:41.883837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c5fd6-5ffe-40e6-b322-5d6867b38297 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:41.884562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967c5fd6-5ffe-40e6-b322-5d6867b38297 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=66c431a7c3ca427b8be80e91bc4dd624, user.id=d5115206a65945c7af922e91bfa9dcaf) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:41.884870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967c5fd6-5ffe-40e6-b322-5d6867b38297 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:41.886412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967c5fd6-5ffe-40e6-b322-5d6867b38297 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:41.892050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 575/1161] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:41 2022] GET /identity/v3/role_assignments?user.id=d5115206a65945c7af922e91bfa9dcaf&scope.project.id=66c431a7c3ca427b8be80e91bc4dd624&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:44.098709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b68945-bb84-4142-807f-e420db7fb06b None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:44.099267 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b68945-bb84-4142-807f-e420db7fb06b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:44.099267 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b68945-bb84-4142-807f-e420db7fb06b None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:44.100587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 587/1162] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:44.105725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dbca13-041e-471e-a781-ecca925dfb87 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:44.106103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dbca13-041e-471e-a781-ecca925dfb87 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:44.106279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dbca13-041e-471e-a781-ecca925dfb87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:44.125072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-14dbca13-041e-471e-a781-ecca925dfb87 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:44.126241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-14dbca13-041e-471e-a781-ecca925dfb87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:44.139229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:44.192505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 576/1163] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:44.464408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b68945-bb84-4142-807f-e420db7fb06b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:44.564439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b68945-bb84-4142-807f-e420db7fb06b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:44.596896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:44.597515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:44.597791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:44.598060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:44.637515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:44.662967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:44.664389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:44.757065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 588/1164] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:44.762941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dbca13-041e-471e-a781-ecca925dfb87 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:44.856841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dbca13-041e-471e-a781-ecca925dfb87 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:44.888770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327c90fa-0036-4f86-9b46-e92783939d14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:44.889411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c90fa-0036-4f86-9b46-e92783939d14 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:44.889688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c90fa-0036-4f86-9b46-e92783939d14 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:44.889955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327c90fa-0036-4f86-9b46-e92783939d14 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:44.896395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327c90fa-0036-4f86-9b46-e92783939d14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:44.897836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327c90fa-0036-4f86-9b46-e92783939d14 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-327c90fa-0036-4f86-9b46-e92783939d14 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:44.909730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:44.911026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:06:44.912167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 577/1165] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:06:44 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:44.916726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:44.982867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:45.011786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf1a8b0-8ccb-4a10-a8e8-ac1b32b73869 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:45.045080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c5a156-17ac-4059-933b-4af03b39bfcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:45.045671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5a156-17ac-4059-933b-4af03b39bfcc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:45.045949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5a156-17ac-4059-933b-4af03b39bfcc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:45.046218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5a156-17ac-4059-933b-4af03b39bfcc None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:45.046846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c5a156-17ac-4059-933b-4af03b39bfcc None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:45.047145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c5a156-17ac-4059-933b-4af03b39bfcc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:45.049050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c5a156-17ac-4059-933b-4af03b39bfcc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:45.056030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 589/1166] 10.222.0.26 () {56 vars in 1115 bytes} [Sat Oct 1 04:06:44 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:47.219805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91e3043-ec9d-4664-a82e-41446ce590b3 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:47.222018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91e3043-ec9d-4664-a82e-41446ce590b3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:47.222018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91e3043-ec9d-4664-a82e-41446ce590b3 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:47.222223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 578/1167] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:47.228422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8985290f-975c-4d40-b9e1-3f20cd3561c6 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:47.228869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8985290f-975c-4d40-b9e1-3f20cd3561c6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:47.229022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8985290f-975c-4d40-b9e1-3f20cd3561c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:47.247849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8985290f-975c-4d40-b9e1-3f20cd3561c6 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:47.249056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8985290f-975c-4d40-b9e1-3f20cd3561c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:47.312665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 590/1168] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:47.577737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91e3043-ec9d-4664-a82e-41446ce590b3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:47.589292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:47.677456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91e3043-ec9d-4664-a82e-41446ce590b3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:47.709476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1141858c-758a-496d-a896-74cdbcdb1545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:47.710060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141858c-758a-496d-a896-74cdbcdb1545 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:47.710334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141858c-758a-496d-a896-74cdbcdb1545 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:47.710598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1141858c-758a-496d-a896-74cdbcdb1545 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:47.711243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1141858c-758a-496d-a896-74cdbcdb1545 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:47.712580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1141858c-758a-496d-a896-74cdbcdb1545 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:47.727894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 579/1169] 10.222.0.26 () {60 vars in 1138 bytes} [Sat Oct 1 04:06:47 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:06:49.949920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d8880-c2fc-4e01-958a-c4edada241ea None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:49.950449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d8880-c2fc-4e01-958a-c4edada241ea None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:49.950449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d8880-c2fc-4e01-958a-c4edada241ea None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:49.951798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 591/1170] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:49.958085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dd300-bcf3-47e6-995e-5e65a46e26fc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:49.958505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dd300-bcf3-47e6-995e-5e65a46e26fc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:49.958693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dd300-bcf3-47e6-995e-5e65a46e26fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:49.976639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ea9dd300-bcf3-47e6-995e-5e65a46e26fc None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:49.977842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea9dd300-bcf3-47e6-995e-5e65a46e26fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:50.036299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:50.042956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 580/1171] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:50.309705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d8880-c2fc-4e01-958a-c4edada241ea None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:50.373606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:50.409149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d8880-c2fc-4e01-958a-c4edada241ea None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:50.441804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e240179-a193-4c22-801f-1634d4cb02db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:50.442422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e240179-a193-4c22-801f-1634d4cb02db None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:50.442701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e240179-a193-4c22-801f-1634d4cb02db None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:50.442974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e240179-a193-4c22-801f-1634d4cb02db None admin] PATH_INFO: `/v3/domains/e41e3b1d5c414e4eb45b27399141645a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:50.452910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e240179-a193-4c22-801f-1634d4cb02db None admin] RBAC: Authorizing `identity:get_domain(domain_id=e41e3b1d5c414e4eb45b27399141645a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:50.454878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e240179-a193-4c22-801f-1634d4cb02db None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:50.456440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 592/1172] 10.222.0.26 () {56 vars in 1183 bytes} [Sat Oct 1 04:06:50 2022] GET /identity/v3/domains/e41e3b1d5c414e4eb45b27399141645a => generated 267 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:50.462193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9dd300-bcf3-47e6-995e-5e65a46e26fc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:50.557977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9dd300-bcf3-47e6-995e-5e65a46e26fc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:50.590696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee2c61c-4608-4b61-8733-f4e3918d24ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:50.591236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee2c61c-4608-4b61-8733-f4e3918d24ed None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:50.591513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee2c61c-4608-4b61-8733-f4e3918d24ed None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:50.591778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee2c61c-4608-4b61-8733-f4e3918d24ed None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:50.592425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee2c61c-4608-4b61-8733-f4e3918d24ed None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:50.593871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee2c61c-4608-4b61-8733-f4e3918d24ed None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:50.619643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 581/1173] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:06:50 2022] POST /identity/v3/projects => generated 361 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:06:52.806578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7273aa-9753-4cf6-acc7-993ab98ac27d None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:52.807093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7273aa-9753-4cf6-acc7-993ab98ac27d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:52.807093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7273aa-9753-4cf6-acc7-993ab98ac27d None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:52.808367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 593/1174] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:52.813799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f1d67-9547-4fac-9167-7aed61e0dd0f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:52.814250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f1d67-9547-4fac-9167-7aed61e0dd0f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:52.814354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720f1d67-9547-4fac-9167-7aed61e0dd0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:52.832530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-720f1d67-9547-4fac-9167-7aed61e0dd0f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:52.833745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-720f1d67-9547-4fac-9167-7aed61e0dd0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:52.877993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:52.899174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 582/1175] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:53.171505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7273aa-9753-4cf6-acc7-993ab98ac27d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:53.271837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7273aa-9753-4cf6-acc7-993ab98ac27d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:53.302553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:53.311695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e52f9c-1c49-4bfa-b66a-a21f4ff89e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:53.312314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e52f9c-1c49-4bfa-b66a-a21f4ff89e17 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:53.312590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e52f9c-1c49-4bfa-b66a-a21f4ff89e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:53.312856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e52f9c-1c49-4bfa-b66a-a21f4ff89e17 None admin] PATH_INFO: `/v3/domains/e41e3b1d5c414e4eb45b27399141645a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:53.314554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e52f9c-1c49-4bfa-b66a-a21f4ff89e17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=e41e3b1d5c414e4eb45b27399141645a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:53.316219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e52f9c-1c49-4bfa-b66a-a21f4ff89e17 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:53.317789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 594/1176] 10.222.0.26 () {56 vars in 1183 bytes} [Sat Oct 1 04:06:53 2022] GET /identity/v3/domains/e41e3b1d5c414e4eb45b27399141645a => generated 267 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:53.323527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f1d67-9547-4fac-9167-7aed61e0dd0f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:53.420609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720f1d67-9547-4fac-9167-7aed61e0dd0f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:53.453283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb482021-4592-4eeb-9028-dd74f4ccc4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:53.453823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb482021-4592-4eeb-9028-dd74f4ccc4f9 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:53.454156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb482021-4592-4eeb-9028-dd74f4ccc4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:53.454430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb482021-4592-4eeb-9028-dd74f4ccc4f9 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:53.455074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb482021-4592-4eeb-9028-dd74f4ccc4f9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:53.456482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb482021-4592-4eeb-9028-dd74f4ccc4f9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:53.476811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 583/1177] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:06:53 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:06:55.647400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4633678d-6b8b-4b01-906f-3d5bae41f59e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:55.647957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4633678d-6b8b-4b01-906f-3d5bae41f59e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:55.647957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4633678d-6b8b-4b01-906f-3d5bae41f59e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:55.649290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 595/1178] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:55.654390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c19b21-328b-40ce-8b9c-5d4378ad6bd4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:55.654769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c19b21-328b-40ce-8b9c-5d4378ad6bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:55.654940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c19b21-328b-40ce-8b9c-5d4378ad6bd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:55.673542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-71c19b21-328b-40ce-8b9c-5d4378ad6bd4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:55.674701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-71c19b21-328b-40ce-8b9c-5d4378ad6bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:55.701464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:55.740092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 584/1179] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:56.010046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4633678d-6b8b-4b01-906f-3d5bae41f59e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:56.109504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4633678d-6b8b-4b01-906f-3d5bae41f59e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:56.142138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:56.142754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:56.143035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:56.143308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:56.153010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:56.158529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:56.159612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:56.160660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 596/1180] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:56.165569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c19b21-328b-40ce-8b9c-5d4378ad6bd4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:56.259317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c19b21-328b-40ce-8b9c-5d4378ad6bd4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:56.291758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:56.292287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:56.292560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:56.292826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:56.293499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:56.293796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:56.295135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:56.303512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 585/1181] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:56.309496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:56.363734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:56.407399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59f4c50-c4ad-46df-a0d5-fb4ce6ea7158 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:56.440536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:56.441201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:56.441484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:56.441758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] PATH_INFO: `/v3/users/3604e8e20a834d9eb8b1866dc2f1227d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:56.454194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] RBAC: Authorizing `identity:get_user(user_id=3604e8e20a834d9eb8b1866dc2f1227d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:56.455616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:56.457231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 597/1182] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:56 2022] GET /identity/v3/users/3604e8e20a834d9eb8b1866dc2f1227d => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:56.462971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:56.490791 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:56.562255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44a4e73-cad9-48fa-821b-189a823bb748 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:56.593645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:56.594243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:56.594616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:56.595420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:56.661646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:56.663085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:56.689685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:56.761014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 586/1183] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:06:56 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:06:56.767104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:56.863920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70daf5f5-d91e-44b1-8168-46f73a8d0be3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:56.876870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:56.897787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280fb07-7bc6-4e3e-aeab-d876e88337ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:56.898402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280fb07-7bc6-4e3e-aeab-d876e88337ad None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:56.898676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280fb07-7bc6-4e3e-aeab-d876e88337ad None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:56.898948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280fb07-7bc6-4e3e-aeab-d876e88337ad None admin] PATH_INFO: `/v3/projects/37da0677941c40ff9582555c50d7f8c0` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:56.900040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280fb07-7bc6-4e3e-aeab-d876e88337ad None admin] RBAC: Authorizing `identity:get_project(project_id=37da0677941c40ff9582555c50d7f8c0)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:56.901481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280fb07-7bc6-4e3e-aeab-d876e88337ad None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:56.903069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 598/1184] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:56 2022] GET /identity/v3/projects/37da0677941c40ff9582555c50d7f8c0 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:56.910162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:57.004356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573eb1e6-6865-4106-ada7-687bd6d9a059 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:57.034126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:57.042059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9214c9d-ac1f-41f4-9628-c991bb286548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:57.042664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9214c9d-ac1f-41f4-9628-c991bb286548 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:57.042946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9214c9d-ac1f-41f4-9628-c991bb286548 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:57.043222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9214c9d-ac1f-41f4-9628-c991bb286548 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:57.043937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9214c9d-ac1f-41f4-9628-c991bb286548 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=37da0677941c40ff9582555c50d7f8c0, user.id=3604e8e20a834d9eb8b1866dc2f1227d) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:57.044244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9214c9d-ac1f-41f4-9628-c991bb286548 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:57.045697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9214c9d-ac1f-41f4-9628-c991bb286548 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:57.051836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 587/1185] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:06:56 2022] GET /identity/v3/role_assignments?user.id=3604e8e20a834d9eb8b1866dc2f1227d&scope.project.id=37da0677941c40ff9582555c50d7f8c0&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:59.149022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eb4fed-ae34-4019-b950-30cd3a04001d None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:59.149558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eb4fed-ae34-4019-b950-30cd3a04001d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:59.149558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eb4fed-ae34-4019-b950-30cd3a04001d None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:59.150918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 599/1186] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:06:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:06:59.156053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a873a7f7-1464-4aa4-984f-0f3579e72bfe None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:59.156487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a873a7f7-1464-4aa4-984f-0f3579e72bfe None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:59.156591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a873a7f7-1464-4aa4-984f-0f3579e72bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:59.175905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a873a7f7-1464-4aa4-984f-0f3579e72bfe None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:06:59.177121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a873a7f7-1464-4aa4-984f-0f3579e72bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:06:59.241736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 588/1187] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:06:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:06:59.506944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eb4fed-ae34-4019-b950-30cd3a04001d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:59.613482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eb4fed-ae34-4019-b950-30cd3a04001d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:59.647943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:59.648570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:59.648845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:59.649147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:59.659516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:06:59.664188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:06:59.665316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:06:59.666165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 600/1188] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:06:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:06:59.671371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a873a7f7-1464-4aa4-984f-0f3579e72bfe None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:59.770134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a873a7f7-1464-4aa4-984f-0f3579e72bfe None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:59.793249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:59.805332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392924ea-7950-45f1-8388-493325baa085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:59.805929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392924ea-7950-45f1-8388-493325baa085 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:59.806205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392924ea-7950-45f1-8388-493325baa085 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:59.806472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392924ea-7950-45f1-8388-493325baa085 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:59.807121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392924ea-7950-45f1-8388-493325baa085 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:06:59.807419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392924ea-7950-45f1-8388-493325baa085 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:59.808755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392924ea-7950-45f1-8388-493325baa085 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:59.816664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 589/1189] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:06:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:59.822557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:06:59.863974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:06:59.919161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaea65b-a937-4a41-a077-85bd9d610031 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:06:59.952279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:06:59.952885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:06:59.953191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:06:59.953464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] PATH_INFO: `/v3/users/3604e8e20a834d9eb8b1866dc2f1227d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:06:59.955256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] RBAC: Authorizing `identity:get_user(user_id=3604e8e20a834d9eb8b1866dc2f1227d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:06:59.956651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:06:59.958212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 601/1190] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:06:59 2022] GET /identity/v3/users/3604e8e20a834d9eb8b1866dc2f1227d => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:06:59.964062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392924ea-7950-45f1-8388-493325baa085 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:00.060242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392924ea-7950-45f1-8388-493325baa085 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:00.092520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13195ef5-fb43-4458-8ad9-6c79680231f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:00.093155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13195ef5-fb43-4458-8ad9-6c79680231f8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:00.093435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13195ef5-fb43-4458-8ad9-6c79680231f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:00.093707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13195ef5-fb43-4458-8ad9-6c79680231f8 None admin] PATH_INFO: `/v3/projects/37da0677941c40ff9582555c50d7f8c0` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:00.094925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13195ef5-fb43-4458-8ad9-6c79680231f8 None admin] RBAC: Authorizing `identity:get_project(project_id=37da0677941c40ff9582555c50d7f8c0)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:07:00.096328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13195ef5-fb43-4458-8ad9-6c79680231f8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:07:00.097925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 590/1191] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:06:59 2022] GET /identity/v3/projects/37da0677941c40ff9582555c50d7f8c0 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:07:00.103892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:00.202680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ae94ff-d162-4203-b3be-33491794d2df None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:00.211933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:00.237120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206a8ad9-4e88-447d-a244-ca1abe5e3e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:00.237766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a8ad9-4e88-447d-a244-ca1abe5e3e43 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:00.238048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a8ad9-4e88-447d-a244-ca1abe5e3e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:00.238325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a8ad9-4e88-447d-a244-ca1abe5e3e43 None admin] PATH_INFO: `/v3/projects/37da0677941c40ff9582555c50d7f8c0/users/3604e8e20a834d9eb8b1866dc2f1227d/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:00.240202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206a8ad9-4e88-447d-a244-ca1abe5e3e43 None admin] RBAC: Authorizing `identity:create_grant(project_id=37da0677941c40ff9582555c50d7f8c0, user_id=3604e8e20a834d9eb8b1866dc2f1227d, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:07:00.241651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206a8ad9-4e88-447d-a244-ca1abe5e3e43 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:07:00.254548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 602/1192] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:07:00 2022] PUT /identity/v3/projects/37da0677941c40ff9582555c50d7f8c0/users/3604e8e20a834d9eb8b1866dc2f1227d/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:07:02.386165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82975c9f-6a93-4ba8-aaef-b72cedd968fe None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:02.386698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82975c9f-6a93-4ba8-aaef-b72cedd968fe None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:02.386698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82975c9f-6a93-4ba8-aaef-b72cedd968fe None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:02.388024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 591/1193] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:07:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:07:02.393920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3688491-4d17-4c79-a9ce-ac011d474d34 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:02.394321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3688491-4d17-4c79-a9ce-ac011d474d34 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:02.394476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3688491-4d17-4c79-a9ce-ac011d474d34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:02.413022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d3688491-4d17-4c79-a9ce-ac011d474d34 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:07:02.414202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d3688491-4d17-4c79-a9ce-ac011d474d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:07:02.478020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 603/1194] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:07:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:07:02.760627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82975c9f-6a93-4ba8-aaef-b72cedd968fe None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:02.814382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:02.860697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82975c9f-6a93-4ba8-aaef-b72cedd968fe None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:02.893539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:02.894160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:02.894443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:02.894715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:02.904432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:07:02.909780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:07:02.910850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:07:02.912075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 592/1195] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:07:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:07:02.916805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3688491-4d17-4c79-a9ce-ac011d474d34 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:03.008353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:03.012596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3688491-4d17-4c79-a9ce-ac011d474d34 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:03.045783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:03.046375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:03.046658 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:03.046928 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:03.048312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:07:03.048614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:07:03.049992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:07:03.057784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 604/1196] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:07:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:07:03.063723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:03.160264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d43a2c-b14c-4706-a478-b43ad8f92343 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:03.172747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:03.193192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:03.193808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:03.194090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:03.194362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] PATH_INFO: `/v3/users/3604e8e20a834d9eb8b1866dc2f1227d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:03.195454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] RBAC: Authorizing `identity:get_user(user_id=3604e8e20a834d9eb8b1866dc2f1227d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:07:03.196869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:07:03.198429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 593/1197] 10.222.0.26 () {56 vars in 1177 bytes} [Sat Oct 1 04:07:03 2022] GET /identity/v3/users/3604e8e20a834d9eb8b1866dc2f1227d => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:07:03.204828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:03.299521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc6c89-634f-4506-a0bf-fbca82d474ed None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:03.331075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:03.331660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:03.331936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:03.332204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:03.357852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:03.398036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:07:03.399456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:07:03.493002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 605/1198] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:07:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 01 04:07:03.499012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:03.596322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11344174-f4d3-4e4d-9b60-e2ddf470d1e3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:03.627870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda607e7-b56d-437e-b9f7-f061009d275a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:03.628487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda607e7-b56d-437e-b9f7-f061009d275a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:03.628763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda607e7-b56d-437e-b9f7-f061009d275a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:03.629061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda607e7-b56d-437e-b9f7-f061009d275a None admin] PATH_INFO: `/v3/projects/37da0677941c40ff9582555c50d7f8c0` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:03.630174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda607e7-b56d-437e-b9f7-f061009d275a None admin] RBAC: Authorizing `identity:get_project(project_id=37da0677941c40ff9582555c50d7f8c0)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:07:03.631574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda607e7-b56d-437e-b9f7-f061009d275a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:07:03.633183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 594/1199] 10.222.0.26 () {56 vars in 1186 bytes} [Sat Oct 1 04:07:03 2022] GET /identity/v3/projects/37da0677941c40ff9582555c50d7f8c0 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:07:03.639330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:07:03.696758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:03.736336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942465d5-61b7-4f69-a7ae-294a944041f7 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:07:03.769379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7d88af-08cf-4124-acd7-071ce3d9ee98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:07:03.769972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7d88af-08cf-4124-acd7-071ce3d9ee98 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:03.770250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7d88af-08cf-4124-acd7-071ce3d9ee98 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:03.770518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7d88af-08cf-4124-acd7-071ce3d9ee98 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:03.771236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7d88af-08cf-4124-acd7-071ce3d9ee98 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=37da0677941c40ff9582555c50d7f8c0, user.id=3604e8e20a834d9eb8b1866dc2f1227d) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:07:03.771544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7d88af-08cf-4124-acd7-071ce3d9ee98 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:07:03.772999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7d88af-08cf-4124-acd7-071ce3d9ee98 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:07:03.778592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 606/1200] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:07:03 2022] GET /identity/v3/role_assignments?user.id=3604e8e20a834d9eb8b1866dc2f1227d&scope.project.id=37da0677941c40ff9582555c50d7f8c0&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:07:13.623753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=76 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:13.776107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=38 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:07:59.825065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751155bf-2b54-4690-a0ae-7a2f8988b702 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:59.825065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751155bf-2b54-4690-a0ae-7a2f8988b702 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:59.825065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751155bf-2b54-4690-a0ae-7a2f8988b702 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:59.827420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 595/1201] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:07:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:07:59.833630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e692e725-5177-4baa-91cc-7519201131d1 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:07:59.834095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e692e725-5177-4baa-91cc-7519201131d1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:07:59.834188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e692e725-5177-4baa-91cc-7519201131d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:07:59.854186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e692e725-5177-4baa-91cc-7519201131d1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:07:59.855500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e692e725-5177-4baa-91cc-7519201131d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:07:59.970048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 607/1202] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:07:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:00.266051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751155bf-2b54-4690-a0ae-7a2f8988b702 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:00.426986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751155bf-2b54-4690-a0ae-7a2f8988b702 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:00.470742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:00.471783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:00.472268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:00.472779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:00.485182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:00.486650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:00.509307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:00.510432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:00.513403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:00.513403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:00.513403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:00.513403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 596/1203] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:00.519883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e692e725-5177-4baa-91cc-7519201131d1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:00.663233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e692e725-5177-4baa-91cc-7519201131d1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:00.697166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:00.703082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebbdefa-bb77-4dc9-a7c6-1f9a373324b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:00.703737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebbdefa-bb77-4dc9-a7c6-1f9a373324b0 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:00.704027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebbdefa-bb77-4dc9-a7c6-1f9a373324b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:00.704309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebbdefa-bb77-4dc9-a7c6-1f9a373324b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:00.705041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebbdefa-bb77-4dc9-a7c6-1f9a373324b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:00.705484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebbdefa-bb77-4dc9-a7c6-1f9a373324b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:00.707400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebbdefa-bb77-4dc9-a7c6-1f9a373324b0 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:00.727159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 608/1204] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:00.730141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:00.811121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:00.875915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d752d2a2-78ed-426a-affa-e34b29728999 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:00.920658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1960d8ce-59de-4bf4-b0d9-83757f076f73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:00.921983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1960d8ce-59de-4bf4-b0d9-83757f076f73 None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:00.922507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1960d8ce-59de-4bf4-b0d9-83757f076f73 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:00.922750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1960d8ce-59de-4bf4-b0d9-83757f076f73 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:00.923416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1960d8ce-59de-4bf4-b0d9-83757f076f73 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:00.924862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1960d8ce-59de-4bf4-b0d9-83757f076f73 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:00.955367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 597/1205] 10.222.0.26 () {60 vars in 1132 bytes} [Sat Oct 1 04:08:00 2022] POST /identity/v3/users => generated 251 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:08:03.528224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3334c16-9506-4d5b-846a-80ea088b6160 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:03.528707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3334c16-9506-4d5b-846a-80ea088b6160 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:03.528800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3334c16-9506-4d5b-846a-80ea088b6160 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:03.530343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 609/1206] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:03.536659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d3f59f-2ae3-421d-8d3a-9e0b063f0895 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:03.537141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d3f59f-2ae3-421d-8d3a-9e0b063f0895 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:03.537322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d3f59f-2ae3-421d-8d3a-9e0b063f0895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:03.561380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d0d3f59f-2ae3-421d-8d3a-9e0b063f0895 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:03.563545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d0d3f59f-2ae3-421d-8d3a-9e0b063f0895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:03.646257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 598/1207] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:03.959375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3334c16-9506-4d5b-846a-80ea088b6160 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:04.076813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3334c16-9506-4d5b-846a-80ea088b6160 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:04.124958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:04.126093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:04.126539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:04.126824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:04.147650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:04.148602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:04.149566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 610/1208] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:08:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:04.154343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d3f59f-2ae3-421d-8d3a-9e0b063f0895 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:04.200752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:04.274433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d3f59f-2ae3-421d-8d3a-9e0b063f0895 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:04.313244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:04.313870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:04.314207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:04.314531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:04.315257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:04.315562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:04.317020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:04.330409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 599/1209] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:08:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:04.331645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:04.393341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:04.452860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5ac86c-5692-4ade-871e-297e8c64d293 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:04.493597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:04.494463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:04.495102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:04.495532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:04.506522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:04.508606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:04.527945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:04.529583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:04.531338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:04.531338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:04.531338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:04.531338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 611/1210] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:04.536448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:04.645828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:04.651285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a333ec-3c24-43fe-931f-f043ebc109bf None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:04.694584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:04.695486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:04.696010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:04.696480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:04.697520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:04.697944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:04.699845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:04.713630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 600/1211] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:04.721274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:04.833387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75082d78-75e2-4320-bba1-b2c5fc157e10 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:04.835080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:04.874101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:04.874889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:04.875265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:04.875638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:04.883786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:04.885383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:08:04.898866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:04.900118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:04.901330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:04.908958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 612/1212] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:04.909756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:05.026890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f6936-ba51-4bb3-9907-30a1afcb2745 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:05.077056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:05.077926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:05.078339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:05.078930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:05.079908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:05.080286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:05.081840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:05.115015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 601/1213] 10.222.0.26 () {56 vars in 1143 bytes} [Sat Oct 1 04:08:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:05.116630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:05.252005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d74d7-16cd-4aa6-b976-f6d634ef321f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:05.298399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:05.299119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:05.299494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:05.299828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:05.337004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:05.398749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:05.400647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:05.536090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 613/1214] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:08:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:08:05.541950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:05.562333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:05.657476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656025c-0fa9-4660-9948-60a715f5b396 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:05.693753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:05.694386 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:05.694670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:05.694945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:05.705479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:05.706922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:05.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:05.725407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:05.726309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:05.726309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:05.726309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:05.727295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 602/1215] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:08:05.732890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:05.800905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:05.841280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954658b5-fb46-4b2a-bc9c-89ac72ca3a98 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:05.876905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:05.877492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:05.877774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:05.878049 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:05.878703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:05.879009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:05.880367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:05.896610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 614/1216] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:05.897878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:05.969720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:06.003506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e277bb6f-6384-4ac0-a999-83dfe6d63be6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:06.040283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:06.040919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:06.041346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:06.041628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:06.052288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:06.054305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:08:06.068528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:06.069671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:06.070699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 603/1217] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:08:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:06.075939 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:06.197020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358f4456-80f7-419f-896e-eed37f2d3ae1 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:06.221034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:06.234621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb64b4d6-3f3e-402d-91d2-b04cfa942202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:06.235222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb64b4d6-3f3e-402d-91d2-b04cfa942202 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:06.235510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb64b4d6-3f3e-402d-91d2-b04cfa942202 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:06.235771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb64b4d6-3f3e-402d-91d2-b04cfa942202 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:06.236445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb64b4d6-3f3e-402d-91d2-b04cfa942202 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:06.236747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb64b4d6-3f3e-402d-91d2-b04cfa942202 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:06.238191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb64b4d6-3f3e-402d-91d2-b04cfa942202 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:06.250427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 615/1218] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:08:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:06.257006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:06.364276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52943198-8b28-4154-8f40-482eb1c9d8f3 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:06.394179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:06.401414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47488fe9-6290-493f-bedf-3c08c08c451b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:06.402019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47488fe9-6290-493f-bedf-3c08c08c451b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:06.402296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47488fe9-6290-493f-bedf-3c08c08c451b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:06.402568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47488fe9-6290-493f-bedf-3c08c08c451b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:06.403374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47488fe9-6290-493f-bedf-3c08c08c451b None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=4302238ce83b4591b3d062ef87517f1c) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:06.403609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47488fe9-6290-493f-bedf-3c08c08c451b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:06.406002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47488fe9-6290-493f-bedf-3c08c08c451b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:06.412228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 604/1219] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:08:06 2022] GET /identity/v3/role_assignments?user.id=4302238ce83b4591b3d062ef87517f1c&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:08.611117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6990f0-614d-4f44-9221-e9b3516c0355 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:08.611663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6990f0-614d-4f44-9221-e9b3516c0355 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:08.611663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6990f0-614d-4f44-9221-e9b3516c0355 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:08.613271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 616/1220] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:08.618721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d644f1-403b-4ede-bc05-e97f1a1e7cdc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:08.619222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d644f1-403b-4ede-bc05-e97f1a1e7cdc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:08.619324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d644f1-403b-4ede-bc05-e97f1a1e7cdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:08.639124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-39d644f1-403b-4ede-bc05-e97f1a1e7cdc None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:08.640632 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-39d644f1-403b-4ede-bc05-e97f1a1e7cdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:08.721668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 605/1221] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:09.050682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6990f0-614d-4f44-9221-e9b3516c0355 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:09.189367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6990f0-614d-4f44-9221-e9b3516c0355 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:09.227539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:09.228195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:09.228471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:09.228744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:09.240464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:08:09.245298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:09.246374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:09.247748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 617/1222] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:08:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:09.253813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d644f1-403b-4ede-bc05-e97f1a1e7cdc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:09.359209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d644f1-403b-4ede-bc05-e97f1a1e7cdc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:09.390475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:09.402568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:09.403192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:09.403574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:09.403819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:09.404552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] RBAC: Adding query filter params (name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:09.404855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:09.406575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:09.415594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 606/1223] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:08:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:09.422372 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:09.478489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:09.542780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb6f01f-6133-4570-943a-7ae251001d4c None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:09.583530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:09.584280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:09.584566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:09.584845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:09.597511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:09.599712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:09.617701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:09.619260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:09.620356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:09.620356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:09.620356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:09.620356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 618/1224] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:09.625147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:09.739581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4ed856-7811-4412-ace6-4c9984165c1e None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:09.777828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:09.778431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:09.778717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:09.778992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:09.779646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:09.780713 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:09.782529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:09.794431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 607/1225] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:09.800903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:09.913168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:09.920084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e2b42-47f3-4ecd-9eec-d985b240436a None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:09.956294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:09.957094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:09.957512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:09.957886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:09.966263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:09.968575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:08:09.982255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:09.984214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:09.985347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:09.989857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 619/1226] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:09.991420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:10.043370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:10.117496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcfa503-b404-451b-890b-cd5bbad8857b None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:10.154085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:10.154695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:10.155090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:10.155448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:10.156416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:10.156737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:10.158174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:10.175832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 608/1227] 10.222.0.26 () {56 vars in 1143 bytes} [Sat Oct 1 04:08:09 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:10.182384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:10.289311 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a45bb6d-6ea2-4e12-ba42-c57b7aacd084 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:10.325442 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:10.326078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:10.326353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:10.326622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:10.338318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:10.339756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:10.355523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:10.356534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:10.357480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:10.357480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:10.357480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:10.357480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 620/1228] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 01 04:08:10.362878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:10.475524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6476af-3b45-41a8-9a07-315d83b3df04 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:10.478324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:10.519618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a2590-e500-44d6-b119-134101082700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:10.520711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a2590-e500-44d6-b119-134101082700 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:10.520711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a2590-e500-44d6-b119-134101082700 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:10.520852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a2590-e500-44d6-b119-134101082700 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:10.521574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a2590-e500-44d6-b119-134101082700 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:10.521912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a2590-e500-44d6-b119-134101082700 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:10.523287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a2590-e500-44d6-b119-134101082700 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:10.538268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 609/1229] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:10.543982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:10.553724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:10.662366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cdf01e-bc4b-44b0-a945-cad23399f916 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:10.701078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:10.701901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:10.702320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:10.702750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:10.710294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:10.711736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:08:10.724154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:10.725360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:10.726345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 621/1230] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:08:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:10.731673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a2590-e500-44d6-b119-134101082700 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:10.846139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a2590-e500-44d6-b119-134101082700 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:10.880461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47810a10-f701-4410-bf3e-4990d0aa8e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:10.881062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47810a10-f701-4410-bf3e-4990d0aa8e02 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:10.881343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47810a10-f701-4410-bf3e-4990d0aa8e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:10.881710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47810a10-f701-4410-bf3e-4990d0aa8e02 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:10.882454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47810a10-f701-4410-bf3e-4990d0aa8e02 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:10.882856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47810a10-f701-4410-bf3e-4990d0aa8e02 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:10.884373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47810a10-f701-4410-bf3e-4990d0aa8e02 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:10.897137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 610/1231] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:08:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:10.903223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:10.969040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:11.004792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c22f35-a2c4-4a2d-bb86-f9e6e397989e None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:11.039482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4027656d-fd13-471d-812e-060a54f47990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:11.040122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4027656d-fd13-471d-812e-060a54f47990 None admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:11.040402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4027656d-fd13-471d-812e-060a54f47990 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:11.040671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4027656d-fd13-471d-812e-060a54f47990 None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/4302238ce83b4591b3d062ef87517f1c/roles/8f8d9a3eff6c47faa94d854a3f42be35` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:11.055389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4027656d-fd13-471d-812e-060a54f47990 None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=4302238ce83b4591b3d062ef87517f1c, role_id=8f8d9a3eff6c47faa94d854a3f42be35)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:11.056942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4027656d-fd13-471d-812e-060a54f47990 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:11.070236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 622/1232] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:08:10 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/4302238ce83b4591b3d062ef87517f1c/roles/8f8d9a3eff6c47faa94d854a3f42be35 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:08:13.404504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6a7ced-76a9-4796-a2ef-d385c89b6eb4 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:13.405983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6a7ced-76a9-4796-a2ef-d385c89b6eb4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:13.405983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6a7ced-76a9-4796-a2ef-d385c89b6eb4 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:13.407454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 611/1233] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:13.412478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6e8cb4-7416-421a-aa30-b4c0a3931d1b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:13.412951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6e8cb4-7416-421a-aa30-b4c0a3931d1b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:13.413095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6e8cb4-7416-421a-aa30-b4c0a3931d1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:13.432602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-1e6e8cb4-7416-421a-aa30-b4c0a3931d1b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:13.434040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1e6e8cb4-7416-421a-aa30-b4c0a3931d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:13.508353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 623/1234] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:13.804954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6a7ced-76a9-4796-a2ef-d385c89b6eb4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:13.836045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:13.925982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6a7ced-76a9-4796-a2ef-d385c89b6eb4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:13.970907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:13.971781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:13.972067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:13.972342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] PATH_INFO: `/v3/roles/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:13.983607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:08:13.988801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:13.990124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:13.991242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 612/1235] 10.222.0.26 () {56 vars in 1102 bytes} [Sat Oct 1 04:08:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:13.996280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6e8cb4-7416-421a-aa30-b4c0a3931d1b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:14.044829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:14.112509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6e8cb4-7416-421a-aa30-b4c0a3931d1b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:14.162463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:14.163080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:14.163364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:14.163639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:14.164312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:14.164619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:14.166026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:14.176333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 624/1236] 10.222.0.26 () {56 vars in 1103 bytes} [Sat Oct 1 04:08:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:14.181381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:14.274647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:14.291355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af32fd6d-9cf9-4f45-907f-ff67680bc3ee None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:14.329681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:14.330321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:14.330608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:14.330886 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:14.341413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:14.343113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:14.357412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:14.358952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:14.364817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:14.364817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:14.364817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:14.364817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 613/1237] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:14.365109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:14.475496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94bd2bb-bd43-4d40-9986-8985cae449e3 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:14.485733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:14.514044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:14.515321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:14.515600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:14.515977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:14.516644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:14.516975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:14.518340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:14.531703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 625/1238] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:14.537884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:14.654296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9829c-1da5-43b4-9194-5d7b1c876c14 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:14.691078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:14.691705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:14.691984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:14.692255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:14.695037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:14.700216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:14.702848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:08:14.715967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:14.717224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:14.718759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 614/1239] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:14.723895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:14.845241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe16d7f-d959-4430-b5f3-912434608385 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:14.880393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:14.881027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:14.881318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:14.881570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:14.882287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:14.882592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:14.884006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:14.904034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 626/1240] 10.222.0.26 () {56 vars in 1143 bytes} [Sat Oct 1 04:08:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:14.910608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:15.019720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dded0c-fd37-4c5b-a485-f08d32d18b8c None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:15.057298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:15.057961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:15.058257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:15.058551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:15.130391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:15.137960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:15.140084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:15.249888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 615/1241] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:08:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 01 04:08:15.256243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:15.286101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:15.364620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b73c6b-ef21-4894-a7a2-cfae5d29df8d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:15.400911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:15.401575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:15.401877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:15.402185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:15.413420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:15.414893 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:15.431272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:15.432286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:15.433616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:15.433616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:15.433616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:15.433616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 627/1242] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:15.439106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:15.550921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:15.557778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc5b86a-7e53-411e-84a5-ea6ff2887ba1 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:15.595587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:15.596281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:15.596604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:15.596900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:15.597586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:15.597891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:15.599465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:15.616147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 616/1243] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:15.622468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:15.716707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:15.742996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b56fdb-adad-4c0f-a08a-36c9ef2b56de None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:15.779115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa5775-1394-4282-9276-30866861b659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:15.780144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa5775-1394-4282-9276-30866861b659 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:15.780694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa5775-1394-4282-9276-30866861b659 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:15.780996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa5775-1394-4282-9276-30866861b659 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:15.788827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fa5775-1394-4282-9276-30866861b659 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:15.790286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fa5775-1394-4282-9276-30866861b659 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-12fa5775-1394-4282-9276-30866861b659 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:08:15.802120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:15.803481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:15.804417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 628/1244] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:08:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:15.809580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:15.939456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8010ae1a-d08e-40df-a6c9-ca38a0b36441 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:15.973818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bbf7c5-b20a-42e0-bc5e-00c36e13c6a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:15.974432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbf7c5-b20a-42e0-bc5e-00c36e13c6a6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:15.974703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbf7c5-b20a-42e0-bc5e-00c36e13c6a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:15.974972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbf7c5-b20a-42e0-bc5e-00c36e13c6a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:15.975654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bbf7c5-b20a-42e0-bc5e-00c36e13c6a6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:15.975957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bbf7c5-b20a-42e0-bc5e-00c36e13c6a6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:15.977407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bbf7c5-b20a-42e0-bc5e-00c36e13c6a6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:15.990583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 617/1245] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:08:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:15.996721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa5775-1394-4282-9276-30866861b659 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:16.118228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa5775-1394-4282-9276-30866861b659 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:16.159365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba915e22-79cc-455e-9ed2-d3a90e4244ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:16.159973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba915e22-79cc-455e-9ed2-d3a90e4244ea None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:16.160249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba915e22-79cc-455e-9ed2-d3a90e4244ea None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:16.160519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba915e22-79cc-455e-9ed2-d3a90e4244ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:16.161298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba915e22-79cc-455e-9ed2-d3a90e4244ea None admin] RBAC: Adding query filter params (role.id=8f8d9a3eff6c47faa94d854a3f42be35, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=4302238ce83b4591b3d062ef87517f1c) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:16.161624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba915e22-79cc-455e-9ed2-d3a90e4244ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:16.163068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba915e22-79cc-455e-9ed2-d3a90e4244ea None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:16.169854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 629/1246] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:08:15 2022] GET /identity/v3/role_assignments?user.id=4302238ce83b4591b3d062ef87517f1c&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=8f8d9a3eff6c47faa94d854a3f42be35 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:18.621833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d723d787-c57a-4d97-9bc8-1e7f1459b2e3 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:18.622501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d723d787-c57a-4d97-9bc8-1e7f1459b2e3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:18.622501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d723d787-c57a-4d97-9bc8-1e7f1459b2e3 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:18.624308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 618/1247] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:18.629572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c847a403-85bb-4f13-b91b-a888fe571b48 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:18.630013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c847a403-85bb-4f13-b91b-a888fe571b48 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:18.630228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c847a403-85bb-4f13-b91b-a888fe571b48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:18.636445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:18.651482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c847a403-85bb-4f13-b91b-a888fe571b48 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:18.652591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c847a403-85bb-4f13-b91b-a888fe571b48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:18.725491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 630/1248] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:08:18.999822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d723d787-c57a-4d97-9bc8-1e7f1459b2e3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:19.018619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:19.109766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d723d787-c57a-4d97-9bc8-1e7f1459b2e3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:19.149124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:19.149776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:19.150057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:19.150331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:19.163161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:08:19.167968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:19.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:19.170086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 619/1249] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:08:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:19.175400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c847a403-85bb-4f13-b91b-a888fe571b48 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:19.275722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c847a403-85bb-4f13-b91b-a888fe571b48 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:19.307091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:19.312995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:19.313594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:19.314101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:19.314564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:19.315311 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:19.315615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:19.317011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:19.325263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 631/1250] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:08:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:19.331536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:19.411759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:19.436160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbbf6f6-7667-4a04-83bf-13b2890e5452 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:19.470365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:19.471000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:19.471282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:19.471560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:19.481619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:19.483384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:19.497738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:19.499416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:19.500497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:19.500497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:19.500497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:19.500497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 620/1251] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:19.505188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:19.601462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8b8c96-7a54-4c9a-b018-ee14e42a9d6d None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:19.634415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:19.635019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:19.635297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:19.635569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:19.636210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:19.636514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:19.637892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:19.649160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 632/1252] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:19.654897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:19.751080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04297b48-24db-4a9c-a69e-90c6d3096cc7 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:19.775255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:19.783378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:19.784289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:19.784289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:19.784510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] PATH_INFO: `/v3/users/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:19.791527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:19.792950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:08:19.804115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:19.805226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:19.806555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 621/1253] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:19 2022] GET /identity/v3/users/placement => generated 87 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:19.812904 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:19.866626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:19.909628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f964870-c344-430e-b789-e9af03fde675 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:19.942660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:19.943247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:19.943524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:19.943800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:19.944513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:19.944816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:19.946318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:19.963375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 633/1254] 10.222.0.26 () {56 vars in 1143 bytes} [Sat Oct 1 04:08:19 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:19.970073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:20.065839 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c978561-7ea5-404b-b234-1f16c16e117a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:20.101479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:20.102178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:20.102460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:20.102732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:20.162626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:20.175063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:20.176496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:20.268934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 622/1255] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:08:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:08:20.274691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:20.370157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca8cd7-cacb-4359-9d84-0ecb08254df9 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:20.382953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:20.403122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:20.403729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:20.404005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:20.404270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:20.414284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:20.415633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:20.430012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:20.431168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:20.432327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:20.432327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:20.432327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:20.432327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 634/1256] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:20.437672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:20.523331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:20.536109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39003a9-e600-41f8-8db6-5cb7a4d04174 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:20.570743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:20.571342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:20.571618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:20.571888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:20.572514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:20.572813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:20.574192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:20.585356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 623/1257] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:20.590943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:20.689522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028fcea3-e258-491e-889f-b336a35332b5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:20.722016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:20.722627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:20.722929 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:20.723202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:20.729546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:20.730954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:08:20.742325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:20.743376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:20.744343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 635/1258] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:08:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:20.749269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:20.845600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eff641-92f5-4af2-9a4a-37d46a7b9065 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:20.878100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee62e7f-7bd9-47e8-bf3c-d4efd9ae1998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:20.878702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee62e7f-7bd9-47e8-bf3c-d4efd9ae1998 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:20.878982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee62e7f-7bd9-47e8-bf3c-d4efd9ae1998 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:20.879253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee62e7f-7bd9-47e8-bf3c-d4efd9ae1998 None admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:20.879932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee62e7f-7bd9-47e8-bf3c-d4efd9ae1998 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:20.880237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee62e7f-7bd9-47e8-bf3c-d4efd9ae1998 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:20.881683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee62e7f-7bd9-47e8-bf3c-d4efd9ae1998 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:20.893136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 624/1259] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:08:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:08:20.899130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:20.924788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:20.994698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed29c332-2da7-4679-a679-d0cdca1a979b None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:21.026753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119e18f3-65b1-4267-becb-f5599f7d2dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:21.027334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119e18f3-65b1-4267-becb-f5599f7d2dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:21.027614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119e18f3-65b1-4267-becb-f5599f7d2dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:21.027882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119e18f3-65b1-4267-becb-f5599f7d2dc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:21.028594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119e18f3-65b1-4267-becb-f5599f7d2dc1 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=4302238ce83b4591b3d062ef87517f1c) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:21.028898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119e18f3-65b1-4267-becb-f5599f7d2dc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:21.030609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119e18f3-65b1-4267-becb-f5599f7d2dc1 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:21.035671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 636/1260] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:08:20 2022] GET /identity/v3/role_assignments?user.id=4302238ce83b4591b3d062ef87517f1c&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:23.194223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99470004-8ab7-4fcf-9aed-4275aae53390 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:23.194789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99470004-8ab7-4fcf-9aed-4275aae53390 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:23.194789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99470004-8ab7-4fcf-9aed-4275aae53390 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:23.196061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 625/1261] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:23.201728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af22e1e4-b2ec-41d6-b723-7ab0be16bce8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:23.202102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af22e1e4-b2ec-41d6-b723-7ab0be16bce8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:23.202284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af22e1e4-b2ec-41d6-b723-7ab0be16bce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:23.220056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-af22e1e4-b2ec-41d6-b723-7ab0be16bce8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:23.221265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-af22e1e4-b2ec-41d6-b723-7ab0be16bce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:23.286086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 637/1262] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:23.557900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99470004-8ab7-4fcf-9aed-4275aae53390 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:23.568774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:23.656597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99470004-8ab7-4fcf-9aed-4275aae53390 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:23.688113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:23.688723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:23.689025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:23.689298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:23.698860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:08:23.704099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:23.705244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:23.706269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 626/1263] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:08:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:23.711175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af22e1e4-b2ec-41d6-b723-7ab0be16bce8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:23.729997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:23.808854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af22e1e4-b2ec-41d6-b723-7ab0be16bce8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:23.842903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:23.843496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:23.843775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:23.844047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:23.844703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:23.845035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:23.846382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:23.854083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 638/1264] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:08:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:23.860522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:23.924358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:23.955879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd71353d-429e-44b4-a860-fca1502d03da None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:23.988824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:23.989464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:23.989740 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:23.990009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:23.999586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:24.001042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:24.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:24.016007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:24.017158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:24.017158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:24.017158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:24.017158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 627/1265] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:24.021825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:24.104476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:24.127542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8acfa51-ec70-4f77-84a1-56bb78881734 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:24.166573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:24.167170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:24.167445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:24.167712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:24.168336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:24.168630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:24.170511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:24.181844 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 639/1266] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:24.188438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:24.283118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0128b53-a3e4-4c76-b67a-b87642d1f250 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:24.303550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:24.318146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:24.318764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:24.319043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:24.319317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:24.325978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:24.327412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:08:24.338398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:24.339505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:24.340498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 628/1267] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:24.345338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:24.445962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636cd592-b12f-4fdf-a915-a20ae65485ba None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:24.479172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:24.479766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:24.480045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:24.480314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:24.481057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:24.481362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:24.482790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:24.491030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:24.499888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 640/1268] 10.222.0.26 () {56 vars in 1143 bytes} [Sat Oct 1 04:08:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:24.505826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:24.602614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e26cef-1227-4dcf-9c37-26b6dda45de4 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:24.634971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:24.635590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:24.635873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:24.636146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:24.647150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:24.648578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:24.663303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:24.664292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:24.665264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:24.665264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:24.665264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:24.665264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 629/1269] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:24.670986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:24.773204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6ca1c4-86db-4394-9ba5-70a3395597f5 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:24.806134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:24.806720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:24.806997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:24.807266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:24.807888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:24.808187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:24.809554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:24.820240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 641/1270] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:24.826451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:24.857397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:24.923312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf1299-dbb2-4c09-b06b-cfddec477970 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:24.955623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:24.956243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:24.956530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:24.956802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:24.963189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:24.964640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:08:24.976032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:24.977114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:24.978210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 630/1271] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:08:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:24.983643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:25.048009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:25.088097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea69fe46-9a93-44aa-8261-530cf9bce57f None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:25.121382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134170f7-5013-4ac7-9565-d746629ab1f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:25.121984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134170f7-5013-4ac7-9565-d746629ab1f7 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:25.122261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134170f7-5013-4ac7-9565-d746629ab1f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:25.122530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134170f7-5013-4ac7-9565-d746629ab1f7 None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:25.123873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134170f7-5013-4ac7-9565-d746629ab1f7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:25.124178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134170f7-5013-4ac7-9565-d746629ab1f7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:25.125634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134170f7-5013-4ac7-9565-d746629ab1f7 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:25.137094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 642/1272] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:08:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:25.143352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:25.231511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:25.240135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e667a3-7468-456f-98a6-cd3ef43baca6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:25.272862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dab4bfc-95d7-4380-a541-047a52df277a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:25.273532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab4bfc-95d7-4380-a541-047a52df277a None admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:25.273812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab4bfc-95d7-4380-a541-047a52df277a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:25.274086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab4bfc-95d7-4380-a541-047a52df277a None admin] PATH_INFO: `/v3/projects/469129ac7df14a70a9cb14818c437800/users/4302238ce83b4591b3d062ef87517f1c/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:25.275935 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dab4bfc-95d7-4380-a541-047a52df277a None admin] RBAC: Authorizing `identity:create_grant(project_id=469129ac7df14a70a9cb14818c437800, user_id=4302238ce83b4591b3d062ef87517f1c, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:25.277389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dab4bfc-95d7-4380-a541-047a52df277a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:25.289303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 631/1273] 10.222.0.26 () {58 vars in 1439 bytes} [Sat Oct 1 04:08:25 2022] PUT /identity/v3/projects/469129ac7df14a70a9cb14818c437800/users/4302238ce83b4591b3d062ef87517f1c/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:08:27.421476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce641b5-995f-43c7-9bd6-8b9b3d6b12d2 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:27.422007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce641b5-995f-43c7-9bd6-8b9b3d6b12d2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:27.422007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce641b5-995f-43c7-9bd6-8b9b3d6b12d2 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:27.423337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 643/1274] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:27.428405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928d659-921d-42e5-9d93-237c9ea95452 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:27.428838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928d659-921d-42e5-9d93-237c9ea95452 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:27.428992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928d659-921d-42e5-9d93-237c9ea95452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:27.447275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b928d659-921d-42e5-9d93-237c9ea95452 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:27.448451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b928d659-921d-42e5-9d93-237c9ea95452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:27.512881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 632/1275] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:27.781619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce641b5-995f-43c7-9bd6-8b9b3d6b12d2 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:27.878244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce641b5-995f-43c7-9bd6-8b9b3d6b12d2 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:27.895251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:27.911633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:27.912240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:27.912513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:27.912780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:27.922809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 01 04:08:27.927398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:27.928507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 01 04:08:27.929736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:27.933253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 644/1276] 10.222.0.26 () {56 vars in 1096 bytes} [Sat Oct 1 04:08:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:27.934357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928d659-921d-42e5-9d93-237c9ea95452 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:28.019027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:28.044146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928d659-921d-42e5-9d93-237c9ea95452 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:28.077542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:28.078137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:28.078413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:28.078681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:28.079331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:28.079630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:28.080987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:28.089841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 633/1277] 10.222.0.26 () {56 vars in 1099 bytes} [Sat Oct 1 04:08:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:28.096071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:28.211067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666609d-1268-4190-8bd9-1300c54d6a49 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:28.243819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:28.244434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:28.244708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:28.245000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:28.254912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:28.256322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:28.269801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:28.270967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:28.272256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:28.272256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:28.272256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:28.272256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 645/1278] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:28.276628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:28.375704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5b276b-b68e-4daf-b2b8-2a2b9c40ab06 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:28.407515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:28.408103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:28.408377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:28.408645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:28.409301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:28.409601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:28.410938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:28.422099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 634/1279] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:28.427722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:28.458566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:28.525766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddebb2a-640c-475a-8ee4-5ad956a56001 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:28.558283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:28.558895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:28.559175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:28.559454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:28.567389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:28.568811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 01 04:08:28.580222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:28.581356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:28.582473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 646/1280] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:28.587136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:28.590855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:28.682422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b979513-2a3e-406c-8ddc-7afbe4701193 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:28.717307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:28.717908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:28.718187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:28.718461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:28.719180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:28.719486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:28.720890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:28.737583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 635/1281] 10.222.0.26 () {56 vars in 1143 bytes} [Sat Oct 1 04:08:28 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:28.743363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:28.831336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:28.840898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6b353-c0cc-482e-8834-8b51872b9017 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:28.874035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:28.874632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:28.874911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:28.875181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:28.940468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:28.941848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:29.027158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:29.038573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 647/1282] 10.222.0.26 () {58 vars in 1282 bytes} [Sat Oct 1 04:08:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 01 04:08:29.044358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:29.105060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:29.142848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7246a9-935a-48cc-aee4-5f44841aaf65 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:29.176111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:29.176726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:29.177033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:29.177303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:29.186779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:29.188195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 01 04:08:29.202406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:29.203992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 01 04:08:29.205301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 01 04:08:29.205301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 01 04:08:29.205301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:29.205301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 636/1283] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:08:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:29.209314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:29.306377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c5427-ece9-4289-b9f3-22987c0cc5dc None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:29.342055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:29.342714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:29.342996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:29.343271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:29.343913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:29.344218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:29.345591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:29.356371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 648/1284] 10.222.0.26 () {56 vars in 1109 bytes} [Sat Oct 1 04:08:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:29.362606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:29.459981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae15add9-6f8e-4820-a2a8-49e086c9795a None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:29.473200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:29.493232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:29.493858 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:29.494141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:29.494414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:29.501314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:29.502749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 01 04:08:29.513814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:29.514963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: ERROR keystone.server.flask.application Oct 01 04:08:29.516260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 637/1285] 10.222.0.26 () {56 vars in 1111 bytes} [Sat Oct 1 04:08:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:29.520638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:29.569726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:29.622503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed295b91-b6ee-449e-9628-8299d3580761 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:29.655511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e011a4-0185-49e0-b909-5b641b095c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:29.656109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e011a4-0185-49e0-b909-5b641b095c8e None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:29.656392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e011a4-0185-49e0-b909-5b641b095c8e None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:29.656665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e011a4-0185-49e0-b909-5b641b095c8e None admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:29.657385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e011a4-0185-49e0-b909-5b641b095c8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:29.657693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e011a4-0185-49e0-b909-5b641b095c8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:29.659106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e011a4-0185-49e0-b909-5b641b095c8e None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:29.671137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 649/1286] 10.222.0.26 () {56 vars in 1148 bytes} [Sat Oct 1 04:08:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:29.677271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:29.771566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee32589-4706-4e89-8d07-4f91d4562378 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:29.803667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef87dbf-7a0c-48e5-9337-8b4aef9962f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:29.804256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef87dbf-7a0c-48e5-9337-8b4aef9962f9 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:29.804533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef87dbf-7a0c-48e5-9337-8b4aef9962f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:29.804803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef87dbf-7a0c-48e5-9337-8b4aef9962f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:29.805537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef87dbf-7a0c-48e5-9337-8b4aef9962f9 None admin] RBAC: Adding query filter params (role.id=1aa2abd4ab14431f8c1b50c02ea27881, scope.project.id=469129ac7df14a70a9cb14818c437800, user.id=4302238ce83b4591b3d062ef87517f1c) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:29.805840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef87dbf-7a0c-48e5-9337-8b4aef9962f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:29.807239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef87dbf-7a0c-48e5-9337-8b4aef9962f9 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:29.813411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 638/1287] 10.222.0.26 () {56 vars in 1374 bytes} [Sat Oct 1 04:08:29 2022] GET /identity/v3/role_assignments?user.id=4302238ce83b4591b3d062ef87517f1c&scope.project.id=469129ac7df14a70a9cb14818c437800&role.id=1aa2abd4ab14431f8c1b50c02ea27881 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:31.983383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b8d75c-7c50-4457-91e1-b7f41c1d9dfc None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:31.983921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b8d75c-7c50-4457-91e1-b7f41c1d9dfc None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:31.983921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b8d75c-7c50-4457-91e1-b7f41c1d9dfc None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:31.985275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 650/1288] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:31.990361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e4e6d-ebb3-4ac8-9587-be4fe2a378a6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:31.990780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e4e6d-ebb3-4ac8-9587-be4fe2a378a6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:31.990920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447e4e6d-ebb3-4ac8-9587-be4fe2a378a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:32.009407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-447e4e6d-ebb3-4ac8-9587-be4fe2a378a6 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:32.010580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-447e4e6d-ebb3-4ac8-9587-be4fe2a378a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:32.039466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:32.076718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 639/1289] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:32.376082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b8d75c-7c50-4457-91e1-b7f41c1d9dfc None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:32.472324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b8d75c-7c50-4457-91e1-b7f41c1d9dfc None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:32.483211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:32.506138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:32.506859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:32.507139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:32.507410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:32.507983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:32.509370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:32.517845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:32.518944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:32.520148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 651/1290] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:08:32 2022] GET /identity/v3/services/placement => generated 90 bytes in 143 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:32.524807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447e4e6d-ebb3-4ac8-9587-be4fe2a378a6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:32.620037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447e4e6d-ebb3-4ac8-9587-be4fe2a378a6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:32.658931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d8b4d5-63ba-41ff-af66-bc1814c374b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:32.659464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d8b4d5-63ba-41ff-af66-bc1814c374b4 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:32.659739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d8b4d5-63ba-41ff-af66-bc1814c374b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:32.660009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d8b4d5-63ba-41ff-af66-bc1814c374b4 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:32.660667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d8b4d5-63ba-41ff-af66-bc1814c374b4 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:32.660996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d8b4d5-63ba-41ff-af66-bc1814c374b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:32.662366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d8b4d5-63ba-41ff-af66-bc1814c374b4 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:32.667050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 640/1291] 10.222.0.26 () {56 vars in 1116 bytes} [Sat Oct 1 04:08:32 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:32.673015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:32.768735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ea3190-a043-4bc2-9820-5c82dd178f25 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:32.800276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7ed79d-ccfd-4e5a-9ab2-2d0dd0989c8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:32.800856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7ed79d-ccfd-4e5a-9ab2-2d0dd0989c8b None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:32.801161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7ed79d-ccfd-4e5a-9ab2-2d0dd0989c8b None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:32.801431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7ed79d-ccfd-4e5a-9ab2-2d0dd0989c8b None admin] PATH_INFO: `/v3/services` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:32.802039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7ed79d-ccfd-4e5a-9ab2-2d0dd0989c8b None admin] RBAC: Adding query filter params (type=placement) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:32.802336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7ed79d-ccfd-4e5a-9ab2-2d0dd0989c8b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:32.803671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7ed79d-ccfd-4e5a-9ab2-2d0dd0989c8b None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:32.807869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 652/1292] 10.222.0.26 () {56 vars in 1116 bytes} [Sat Oct 1 04:08:32 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:34.959889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50d04ee-94d2-4a57-8a5f-dec3a91d3482 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:34.960407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50d04ee-94d2-4a57-8a5f-dec3a91d3482 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:34.960524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50d04ee-94d2-4a57-8a5f-dec3a91d3482 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:34.961787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 641/1293] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:34.967580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebf0c6-26b3-4021-8139-974ef359e663 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:34.967986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebf0c6-26b3-4021-8139-974ef359e663 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:34.968140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebf0c6-26b3-4021-8139-974ef359e663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:34.986512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-44ebf0c6-26b3-4021-8139-974ef359e663 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:34.987694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-44ebf0c6-26b3-4021-8139-974ef359e663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:35.016004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:35.052762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 653/1294] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 01 04:08:35.322674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50d04ee-94d2-4a57-8a5f-dec3a91d3482 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:35.352698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:35.431494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50d04ee-94d2-4a57-8a5f-dec3a91d3482 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:35.465081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4490a71c-ed39-42f4-990b-038c90bc94dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:35.465693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4490a71c-ed39-42f4-990b-038c90bc94dd None admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:35.465953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4490a71c-ed39-42f4-990b-038c90bc94dd None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:35.466262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4490a71c-ed39-42f4-990b-038c90bc94dd None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:35.466894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4490a71c-ed39-42f4-990b-038c90bc94dd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:35.468236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4490a71c-ed39-42f4-990b-038c90bc94dd None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:35.482492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 642/1295] 10.222.0.26 () {60 vars in 1141 bytes} [Sat Oct 1 04:08:35 2022] POST /identity/v3/services => generated 246 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:08:37.621037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e63b471-74d0-483b-aac4-c661c9382cc1 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:37.621559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e63b471-74d0-483b-aac4-c661c9382cc1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:37.621559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e63b471-74d0-483b-aac4-c661c9382cc1 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:37.623477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 654/1296] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:37.628695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303255fb-4df9-47b9-ba1e-a226a0692647 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:37.629140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303255fb-4df9-47b9-ba1e-a226a0692647 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:37.629288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303255fb-4df9-47b9-ba1e-a226a0692647 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:37.647231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-303255fb-4df9-47b9-ba1e-a226a0692647 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:37.648410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-303255fb-4df9-47b9-ba1e-a226a0692647 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:37.719764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 643/1297] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:37 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:37.990857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e63b471-74d0-483b-aac4-c661c9382cc1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:38.090376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e63b471-74d0-483b-aac4-c661c9382cc1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:38.126544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:38.127195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:38.127404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:38.127668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] PATH_INFO: `/v3/services/placement` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:38.128241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:38.129626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:38.139045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:38.140665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:38.141833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 655/1298] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:08:37 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:38.146376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303255fb-4df9-47b9-ba1e-a226a0692647 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:38.166936 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:38.248984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303255fb-4df9-47b9-ba1e-a226a0692647 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:38.282476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:38.283077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:38.283357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:38.283630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:38.284251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:38.284554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:38.285934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:38.290856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 644/1299] 10.222.0.26 () {56 vars in 1116 bytes} [Sat Oct 1 04:08:38 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:38.297617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:38.328396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:38.393875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625739cf-2b81-4c41-b266-a52c5c60a6ee None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:38.425897 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c680ca1-9519-4585-83f4-8f94000e2da3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:38.426478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c680ca1-9519-4585-83f4-8f94000e2da3 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:38.426753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c680ca1-9519-4585-83f4-8f94000e2da3 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:38.427018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c680ca1-9519-4585-83f4-8f94000e2da3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:38.427701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c680ca1-9519-4585-83f4-8f94000e2da3 None admin] RBAC: Adding query filter params (interface=public, service_id=c6ea0844af03422b91acc8e58f8f3980, region_id=RegionOne) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:38.428003 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c680ca1-9519-4585-83f4-8f94000e2da3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:38.429367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c680ca1-9519-4585-83f4-8f94000e2da3 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:38.434092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 656/1300] 10.222.0.26 () {56 vars in 1251 bytes} [Sat Oct 1 04:08:38 2022] GET /identity/v3/endpoints?service_id=c6ea0844af03422b91acc8e58f8f3980&interface=public®ion_id=RegionOne => generated 194 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:38.439831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:38.519517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:38.538610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306e6573-e8be-45b6-a419-7c51652179c6 None admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:38.572662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168726a6-a12e-4791-ba89-ebc54038d6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:38.573393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168726a6-a12e-4791-ba89-ebc54038d6c8 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:38.573721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168726a6-a12e-4791-ba89-ebc54038d6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:38.573997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168726a6-a12e-4791-ba89-ebc54038d6c8 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:38.574658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168726a6-a12e-4791-ba89-ebc54038d6c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:38.576049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168726a6-a12e-4791-ba89-ebc54038d6c8 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:38.582623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 645/1301] 10.222.0.26 () {56 vars in 1087 bytes} [Sat Oct 1 04:08:38 2022] GET /identity/v3/services => generated 2191 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:08:40.730501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a512b6e-93fa-4a9a-8087-f905c4985858 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:40.731021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a512b6e-93fa-4a9a-8087-f905c4985858 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:40.731109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a512b6e-93fa-4a9a-8087-f905c4985858 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:40.732904 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 657/1302] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:08:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:08:40.738212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c3fb5c-5183-4d48-b698-0290ca48377e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:40.738579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c3fb5c-5183-4d48-b698-0290ca48377e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:40.738760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c3fb5c-5183-4d48-b698-0290ca48377e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:40.756962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-10c3fb5c-5183-4d48-b698-0290ca48377e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:08:40.758148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-10c3fb5c-5183-4d48-b698-0290ca48377e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:08:40.826311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 646/1303] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:08:40 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 01 04:08:41.133870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a512b6e-93fa-4a9a-8087-f905c4985858 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:41.231721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:41.239186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a512b6e-93fa-4a9a-8087-f905c4985858 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:41.273768 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:41.274382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:41.274664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:41.274930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:41.275515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:41.276854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 01 04:08:41.287131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 01 04:08:41.288340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.get_or_create( Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application with Lock( Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self._enter() Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application return self.creator() Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application created_value = creator( Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: ERROR keystone.server.flask.application Oct 01 04:08:41.289258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 658/1304] 10.222.0.26 () {56 vars in 1117 bytes} [Sat Oct 1 04:08:41 2022] GET /identity/v3/services/placement => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 01 04:08:41.295184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c3fb5c-5183-4d48-b698-0290ca48377e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:41.378163 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:41.402425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c3fb5c-5183-4d48-b698-0290ca48377e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:41.438068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f348a79-2132-482c-a678-8503877f54e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:41.438677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f348a79-2132-482c-a678-8503877f54e6 None admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:41.438957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f348a79-2132-482c-a678-8503877f54e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:41.439230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f348a79-2132-482c-a678-8503877f54e6 None admin] PATH_INFO: `/v3/services` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:41.439857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f348a79-2132-482c-a678-8503877f54e6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:08:41.440159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f348a79-2132-482c-a678-8503877f54e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:41.441558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f348a79-2132-482c-a678-8503877f54e6 None admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:41.446735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 647/1305] 10.222.0.26 () {56 vars in 1116 bytes} [Sat Oct 1 04:08:41 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:08:41.453224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:08:41.564196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42c67af-1113-494f-97f4-94bc0650e096 None admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:08:41.603503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab4fdb7-dd14-4b9e-b042-6def10a8dcfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:08:41.604105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab4fdb7-dd14-4b9e-b042-6def10a8dcfe None admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:08:41.604385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab4fdb7-dd14-4b9e-b042-6def10a8dcfe None admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:08:41.604654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab4fdb7-dd14-4b9e-b042-6def10a8dcfe None admin] PATH_INFO: `/v3/endpoints` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:08:41.605505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab4fdb7-dd14-4b9e-b042-6def10a8dcfe None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:08:41.607956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab4fdb7-dd14-4b9e-b042-6def10a8dcfe None admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:08:41.629595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 659/1306] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:08:41 2022] POST /identity/v3/endpoints => generated 331 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:08:51.443648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=30 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:08:51.624697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:46.269589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d62c862-155e-4494-849a-25837a601247 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:46.270166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d62c862-155e-4494-849a-25837a601247 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:46.270166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d62c862-155e-4494-849a-25837a601247 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:46.271465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 648/1307] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:09:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:09:46.276952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8956b-f101-4d79-bdc9-8534313ecf48 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:46.277762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8956b-f101-4d79-bdc9-8534313ecf48 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:46.278043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8956b-f101-4d79-bdc9-8534313ecf48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:46.372434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-87e8956b-f101-4d79-bdc9-8534313ecf48 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:09:46.373598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-87e8956b-f101-4d79-bdc9-8534313ecf48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:46.491094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 660/1308] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:09:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:09:46.794914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224cb8f-f85b-4839-b57f-4147823d8163 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:46.795199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224cb8f-f85b-4839-b57f-4147823d8163 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:46.795635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224cb8f-f85b-4839-b57f-4147823d8163 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:46.797259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 649/1309] 10.222.0.26 () {54 vars in 964 bytes} [Sat Oct 1 04:09:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:09:46.803470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d89b2-4ff6-44b6-b565-7a90420547dd None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:46.803828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d89b2-4ff6-44b6-b565-7a90420547dd None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:46.804172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d89b2-4ff6-44b6-b565-7a90420547dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:46.890136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d13d89b2-4ff6-44b6-b565-7a90420547dd None None] MFA Rules not processed for user `16aab142c0a7445483caf19178d56062`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:09:46.891354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d13d89b2-4ff6-44b6-b565-7a90420547dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:46.913489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:46.979365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 661/1310] 10.222.0.26 () {58 vars in 1061 bytes} [Sat Oct 1 04:09:46 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:09:46.988966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224cb8f-f85b-4839-b57f-4147823d8163 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:09:46.990771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e224cb8f-f85b-4839-b57f-4147823d8163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:47.100712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224cb8f-f85b-4839-b57f-4147823d8163 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:09:47.179860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453db0c0-56aa-406e-bec9-96ccff05760a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:09:47.180472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453db0c0-56aa-406e-bec9-96ccff05760a service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:47.180745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453db0c0-56aa-406e-bec9-96ccff05760a service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:47.181501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453db0c0-56aa-406e-bec9-96ccff05760a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:47.182642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-453db0c0-56aa-406e-bec9-96ccff05760a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:47.238608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:47.296999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453db0c0-56aa-406e-bec9-96ccff05760a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:09:47.298448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453db0c0-56aa-406e-bec9-96ccff05760a service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:09:47.317731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 650/1311] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:09:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:09:49.777821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6822df2-8478-4be7-bc09-7ec3b035f42b None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:49.778667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6822df2-8478-4be7-bc09-7ec3b035f42b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:49.778667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6822df2-8478-4be7-bc09-7ec3b035f42b None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:49.780097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 662/1312] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:09:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:09:49.785283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a5863-9a4b-4f5e-bf95-9d001c7d59aa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:49.785687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a5863-9a4b-4f5e-bf95-9d001c7d59aa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:49.785956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a5863-9a4b-4f5e-bf95-9d001c7d59aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:49.856559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-192a5863-9a4b-4f5e-bf95-9d001c7d59aa None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:09:49.857781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-192a5863-9a4b-4f5e-bf95-9d001c7d59aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:49.952796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 651/1313] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:09:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:09:50.240464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6822df2-8478-4be7-bc09-7ec3b035f42b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:09:50.242044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d6822df2-8478-4be7-bc09-7ec3b035f42b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:50.353661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6822df2-8478-4be7-bc09-7ec3b035f42b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:09:50.426905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:50.429067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355eb93d-6022-47e7-b0bc-b38e38b68637 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:09:50.429673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355eb93d-6022-47e7-b0bc-b38e38b68637 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:50.429954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355eb93d-6022-47e7-b0bc-b38e38b68637 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:50.430387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355eb93d-6022-47e7-b0bc-b38e38b68637 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:50.433251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-355eb93d-6022-47e7-b0bc-b38e38b68637 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:50.545157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355eb93d-6022-47e7-b0bc-b38e38b68637 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:09:50.546588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355eb93d-6022-47e7-b0bc-b38e38b68637 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:09:50.564384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 663/1314] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:09:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:09:53.063633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858611fe-9fed-4483-960e-901c87941e8d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:53.064199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858611fe-9fed-4483-960e-901c87941e8d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:53.064199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858611fe-9fed-4483-960e-901c87941e8d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:53.065552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 652/1315] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:09:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:09:53.070823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476f894-6a1c-422a-a53b-439448f270cb None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:53.071266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476f894-6a1c-422a-a53b-439448f270cb None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:53.071968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476f894-6a1c-422a-a53b-439448f270cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:53.138162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b476f894-6a1c-422a-a53b-439448f270cb None None] MFA Rules not processed for user `7ebc4b5874764e32b082febc76d09d5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:09:53.139390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b476f894-6a1c-422a-a53b-439448f270cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:53.189963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:53.233447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 664/1316] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:09:53 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:09:53.527900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db17416-e679-4cc4-be1f-4992d4a413a5 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:53.528446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db17416-e679-4cc4-be1f-4992d4a413a5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:53.528731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db17416-e679-4cc4-be1f-4992d4a413a5 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:53.530101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 653/1317] 10.222.0.26 () {54 vars in 964 bytes} [Sat Oct 1 04:09:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:09:53.535674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2b8c6-f00b-444e-b56c-b9aab0870251 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:53.535962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2b8c6-f00b-444e-b56c-b9aab0870251 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:53.536239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2b8c6-f00b-444e-b56c-b9aab0870251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:53.610616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-68a2b8c6-f00b-444e-b56c-b9aab0870251 None None] MFA Rules not processed for user `16aab142c0a7445483caf19178d56062`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:09:53.611816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-68a2b8c6-f00b-444e-b56c-b9aab0870251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:53.686158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 665/1318] 10.222.0.26 () {58 vars in 1061 bytes} [Sat Oct 1 04:09:53 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:09:53.695086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db17416-e679-4cc4-be1f-4992d4a413a5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:09:53.696828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5db17416-e679-4cc4-be1f-4992d4a413a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:53.707084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:53.799565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db17416-e679-4cc4-be1f-4992d4a413a5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:09:53.870778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fac809c-6762-4058-a867-d46387c142c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:09:53.871376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac809c-6762-4058-a867-d46387c142c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:53.871653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac809c-6762-4058-a867-d46387c142c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:53.871925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac809c-6762-4058-a867-d46387c142c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:53.873517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4fac809c-6762-4058-a867-d46387c142c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:53.933965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:53.981229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fac809c-6762-4058-a867-d46387c142c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:09:53.982660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fac809c-6762-4058-a867-d46387c142c4 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:09:54.000307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 654/1319] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:09:53 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:09:57.064155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391dea4-0a0f-43d1-83bf-8636d54d7d61 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:57.064715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391dea4-0a0f-43d1-83bf-8636d54d7d61 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:57.064715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391dea4-0a0f-43d1-83bf-8636d54d7d61 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:57.066066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 666/1320] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:09:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:09:57.071981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0ca211-5f78-47b4-b2ca-98dd4322e702 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:57.072358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0ca211-5f78-47b4-b2ca-98dd4322e702 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:57.072540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0ca211-5f78-47b4-b2ca-98dd4322e702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:57.147429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cb0ca211-5f78-47b4-b2ca-98dd4322e702 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:09:57.148662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cb0ca211-5f78-47b4-b2ca-98dd4322e702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:57.243654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 655/1321] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:09:57 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:09:57.534812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d391dea4-0a0f-43d1-83bf-8636d54d7d61 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:09:57.536351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d391dea4-0a0f-43d1-83bf-8636d54d7d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:57.586326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:57.640081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d391dea4-0a0f-43d1-83bf-8636d54d7d61 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:09:57.712315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d306d55-daba-44a3-a0c0-b554c89cc934 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:09:57.712906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d306d55-daba-44a3-a0c0-b554c89cc934 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:09:57.713216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d306d55-daba-44a3-a0c0-b554c89cc934 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:09:57.713489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d306d55-daba-44a3-a0c0-b554c89cc934 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:09:57.715078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3d306d55-daba-44a3-a0c0-b554c89cc934 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:09:57.811870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:09:57.826324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d306d55-daba-44a3-a0c0-b554c89cc934 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:09:57.827774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d306d55-daba-44a3-a0c0-b554c89cc934 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:09:57.846018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 667/1322] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:09:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:00.714896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7b4ff3-9f7c-478f-a165-b6d726ee1662 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:00.715432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7b4ff3-9f7c-478f-a165-b6d726ee1662 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:00.715545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7b4ff3-9f7c-478f-a165-b6d726ee1662 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:00.720970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 656/1323] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:00.723109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5aacd-864d-4161-8f03-cf4ef91703fa None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:00.723582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5aacd-864d-4161-8f03-cf4ef91703fa None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:00.723676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5aacd-864d-4161-8f03-cf4ef91703fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:00.801614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5dd5aacd-864d-4161-8f03-cf4ef91703fa None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:00.802749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5dd5aacd-864d-4161-8f03-cf4ef91703fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:00.914422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 668/1324] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:01.200110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7b4ff3-9f7c-478f-a165-b6d726ee1662 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:01.201674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fc7b4ff3-9f7c-478f-a165-b6d726ee1662 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:01.223542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:01.308717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7b4ff3-9f7c-478f-a165-b6d726ee1662 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:01.382647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76383f10-061d-4ca8-9df6-d879093f64b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:01.383253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76383f10-061d-4ca8-9df6-d879093f64b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:01.384214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76383f10-061d-4ca8-9df6-d879093f64b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:01.384496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76383f10-061d-4ca8-9df6-d879093f64b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:01.386134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-76383f10-061d-4ca8-9df6-d879093f64b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:01.457940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:01.496738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76383f10-061d-4ca8-9df6-d879093f64b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:01.498217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76383f10-061d-4ca8-9df6-d879093f64b7 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:01.517328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 657/1325] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:03.756259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f6d947-ccbe-4213-9627-d39c7393d1b2 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:03.756835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f6d947-ccbe-4213-9627-d39c7393d1b2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:03.756835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f6d947-ccbe-4213-9627-d39c7393d1b2 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:03.758412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 669/1326] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:03.763937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf1eec-116d-4fc2-a259-4fcf9438111a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:03.764331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf1eec-116d-4fc2-a259-4fcf9438111a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:03.764494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf1eec-116d-4fc2-a259-4fcf9438111a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:03.839047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6caf1eec-116d-4fc2-a259-4fcf9438111a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:03.840278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6caf1eec-116d-4fc2-a259-4fcf9438111a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:03.936026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 658/1327] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:04.217209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f6d947-ccbe-4213-9627-d39c7393d1b2 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:04.218733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-26f6d947-ccbe-4213-9627-d39c7393d1b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:04.279651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:04.324492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f6d947-ccbe-4213-9627-d39c7393d1b2 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:04.400544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4945c48b-f4f0-41a7-b816-6ffc81582b13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:04.401254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4945c48b-f4f0-41a7-b816-6ffc81582b13 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:04.401536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4945c48b-f4f0-41a7-b816-6ffc81582b13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:04.401813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4945c48b-f4f0-41a7-b816-6ffc81582b13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:04.403692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4945c48b-f4f0-41a7-b816-6ffc81582b13 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:04.510322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:04.512774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4945c48b-f4f0-41a7-b816-6ffc81582b13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:04.514304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4945c48b-f4f0-41a7-b816-6ffc81582b13 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:04.534960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 670/1328] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:06.764260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b0ecf-6844-420b-92e1-5801e1034387 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:06.764759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b0ecf-6844-420b-92e1-5801e1034387 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:06.764759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b0ecf-6844-420b-92e1-5801e1034387 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:06.766078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 659/1329] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:06.771458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e46fa7e-82fb-4221-8e43-8f607123d89f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:06.771867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e46fa7e-82fb-4221-8e43-8f607123d89f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:06.772166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e46fa7e-82fb-4221-8e43-8f607123d89f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:06.847231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7e46fa7e-82fb-4221-8e43-8f607123d89f None None] MFA Rules not processed for user `7ebc4b5874764e32b082febc76d09d5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:06.848357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7e46fa7e-82fb-4221-8e43-8f607123d89f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:06.949433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 671/1330] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:06 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:07.241411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832b0ecf-6844-420b-92e1-5801e1034387 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:07.243593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-832b0ecf-6844-420b-92e1-5801e1034387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:07.276776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:07.361220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832b0ecf-6844-420b-92e1-5801e1034387 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:07.442534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b45e057-8dff-4703-aa24-0ee80ae34bdb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:07.443240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b45e057-8dff-4703-aa24-0ee80ae34bdb service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:07.443420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b45e057-8dff-4703-aa24-0ee80ae34bdb service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:07.443698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b45e057-8dff-4703-aa24-0ee80ae34bdb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:07.445314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6b45e057-8dff-4703-aa24-0ee80ae34bdb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:07.532619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:07.569154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b45e057-8dff-4703-aa24-0ee80ae34bdb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:07.571486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b45e057-8dff-4703-aa24-0ee80ae34bdb service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:07.591580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 660/1331] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:07 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:09.981189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7163b4-2735-4b72-9e75-4727c6b9cb31 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:09.981738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7163b4-2735-4b72-9e75-4727c6b9cb31 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:09.981738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7163b4-2735-4b72-9e75-4727c6b9cb31 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:09.983086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 672/1332] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:09.988222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48d3bc-7f00-4534-927a-9fe588cd91ed None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:09.988663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48d3bc-7f00-4534-927a-9fe588cd91ed None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:09.988795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48d3bc-7f00-4534-927a-9fe588cd91ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:10.052747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8e48d3bc-7f00-4534-927a-9fe588cd91ed None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:10.053967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8e48d3bc-7f00-4534-927a-9fe588cd91ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:10.148263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 661/1333] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:10.437071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7163b4-2735-4b72-9e75-4727c6b9cb31 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:10.438595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ef7163b4-2735-4b72-9e75-4727c6b9cb31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:10.507097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:10.541398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7163b4-2735-4b72-9e75-4727c6b9cb31 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:10.614002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba74fcc0-26f8-466d-9f72-779d39e55801 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:10.614604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74fcc0-26f8-466d-9f72-779d39e55801 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:10.614884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74fcc0-26f8-466d-9f72-779d39e55801 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:10.615156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74fcc0-26f8-466d-9f72-779d39e55801 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:10.616748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ba74fcc0-26f8-466d-9f72-779d39e55801 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:10.722132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba74fcc0-26f8-466d-9f72-779d39e55801 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:10.723475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba74fcc0-26f8-466d-9f72-779d39e55801 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:10.733600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:10.743133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 673/1334] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:12.934969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67aa71-3a34-4a1d-91e4-a33019098878 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:12.935489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67aa71-3a34-4a1d-91e4-a33019098878 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:12.935489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc67aa71-3a34-4a1d-91e4-a33019098878 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:12.936825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 662/1335] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:12.942144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1f72c-3374-4e5d-b4bc-dcfae2503bf8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:12.942586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1f72c-3374-4e5d-b4bc-dcfae2503bf8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:12.942681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1f72c-3374-4e5d-b4bc-dcfae2503bf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:13.008765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2ad1f72c-3374-4e5d-b4bc-dcfae2503bf8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:13.010000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2ad1f72c-3374-4e5d-b4bc-dcfae2503bf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:13.106851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 674/1336] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:13.407368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc67aa71-3a34-4a1d-91e4-a33019098878 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:13.408908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cc67aa71-3a34-4a1d-91e4-a33019098878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:13.450455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:13.512719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc67aa71-3a34-4a1d-91e4-a33019098878 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:13.585164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b81f596-a6d0-421b-8ddb-07a140bcd97a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:13.585698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b81f596-a6d0-421b-8ddb-07a140bcd97a service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:13.586114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b81f596-a6d0-421b-8ddb-07a140bcd97a service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:13.586389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b81f596-a6d0-421b-8ddb-07a140bcd97a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:13.587946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0b81f596-a6d0-421b-8ddb-07a140bcd97a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:13.668939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:13.694311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b81f596-a6d0-421b-8ddb-07a140bcd97a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:13.696513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b81f596-a6d0-421b-8ddb-07a140bcd97a service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:13.714212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 663/1337] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:16.186132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8147bdf5-a7d8-4d63-8ec4-117f98bc86ee None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:16.186587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8147bdf5-a7d8-4d63-8ec4-117f98bc86ee None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:16.186706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8147bdf5-a7d8-4d63-8ec4-117f98bc86ee None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:16.188033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 675/1338] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:16.193395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f80360-1e19-4e0e-9157-0f46e4042758 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:16.193801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f80360-1e19-4e0e-9157-0f46e4042758 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:16.193951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f80360-1e19-4e0e-9157-0f46e4042758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:16.268180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-51f80360-1e19-4e0e-9157-0f46e4042758 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:16.269416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-51f80360-1e19-4e0e-9157-0f46e4042758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:16.364971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 664/1339] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:16 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:16.649853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8147bdf5-a7d8-4d63-8ec4-117f98bc86ee None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:16.651371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8147bdf5-a7d8-4d63-8ec4-117f98bc86ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:16.727070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:16.754062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8147bdf5-a7d8-4d63-8ec4-117f98bc86ee None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:16.826807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c05944d-19ec-4831-91b5-3cdbf3f70045 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:16.827336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05944d-19ec-4831-91b5-3cdbf3f70045 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:16.827617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05944d-19ec-4831-91b5-3cdbf3f70045 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:16.827888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c05944d-19ec-4831-91b5-3cdbf3f70045 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:16.829491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7c05944d-19ec-4831-91b5-3cdbf3f70045 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:16.936944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05944d-19ec-4831-91b5-3cdbf3f70045 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:16.938382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c05944d-19ec-4831-91b5-3cdbf3f70045 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:16.956341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 676/1340] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:16 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:21.109342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7fdb95-75a6-447d-b30f-03b977f2cf3d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:21.109919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7fdb95-75a6-447d-b30f-03b977f2cf3d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:21.109919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7fdb95-75a6-447d-b30f-03b977f2cf3d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:21.111228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 665/1341] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:21.117316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7899e-74cd-4f3f-84f9-8cf433aad96d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:21.118339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7899e-74cd-4f3f-84f9-8cf433aad96d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:21.118339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d7899e-74cd-4f3f-84f9-8cf433aad96d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:21.123508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:21.185843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-54d7899e-74cd-4f3f-84f9-8cf433aad96d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:21.187047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-54d7899e-74cd-4f3f-84f9-8cf433aad96d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:21.292454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 677/1342] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:10:21.578753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7fdb95-75a6-447d-b30f-03b977f2cf3d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:21.580356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1e7fdb95-75a6-447d-b30f-03b977f2cf3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:21.627879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:21.682089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7fdb95-75a6-447d-b30f-03b977f2cf3d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:21.754245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb2d989-444b-4a41-8717-5fccd57af76b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:21.754855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2d989-444b-4a41-8717-5fccd57af76b service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:21.755137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2d989-444b-4a41-8717-5fccd57af76b service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:21.755408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2d989-444b-4a41-8717-5fccd57af76b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:21.757025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0fb2d989-444b-4a41-8717-5fccd57af76b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:21.850145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:21.867651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb2d989-444b-4a41-8717-5fccd57af76b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:21.869114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb2d989-444b-4a41-8717-5fccd57af76b service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:21.888674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 666/1343] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:24.753124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e65690d-2a05-4a8d-8c80-e58d2c2d351e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:24.753675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e65690d-2a05-4a8d-8c80-e58d2c2d351e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:24.753675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e65690d-2a05-4a8d-8c80-e58d2c2d351e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:24.754984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 678/1344] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:24.760287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda939f6-d465-4751-8b7c-b02e97edb6ea None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:24.760704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda939f6-d465-4751-8b7c-b02e97edb6ea None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:24.760901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda939f6-d465-4751-8b7c-b02e97edb6ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:24.827893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fda939f6-d465-4751-8b7c-b02e97edb6ea None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:24.829114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fda939f6-d465-4751-8b7c-b02e97edb6ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:24.932229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 667/1345] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:24 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:25.217885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e65690d-2a05-4a8d-8c80-e58d2c2d351e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:25.219387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6e65690d-2a05-4a8d-8c80-e58d2c2d351e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:25.307664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:25.324911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e65690d-2a05-4a8d-8c80-e58d2c2d351e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:25.397858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c563ee-9c6e-4351-bc04-fb73eca459c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:25.398384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c563ee-9c6e-4351-bc04-fb73eca459c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:25.398662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c563ee-9c6e-4351-bc04-fb73eca459c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:25.398931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c563ee-9c6e-4351-bc04-fb73eca459c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:25.400547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-67c563ee-9c6e-4351-bc04-fb73eca459c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:25.514757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c563ee-9c6e-4351-bc04-fb73eca459c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:25.516194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c563ee-9c6e-4351-bc04-fb73eca459c7 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:25.534733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 679/1346] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:25 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:29.680427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780eb8e3-7bc7-4f11-acc7-e26802297914 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:29.681000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780eb8e3-7bc7-4f11-acc7-e26802297914 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:29.681000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780eb8e3-7bc7-4f11-acc7-e26802297914 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:29.682533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 668/1347] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:29.687798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf46a2-e3ae-4355-9cc2-ec49b2c0b86f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:29.688313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf46a2-e3ae-4355-9cc2-ec49b2c0b86f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:29.688398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf46a2-e3ae-4355-9cc2-ec49b2c0b86f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:29.705317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:29.763173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2adf46a2-e3ae-4355-9cc2-ec49b2c0b86f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:29.764591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2adf46a2-e3ae-4355-9cc2-ec49b2c0b86f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:29.867665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 680/1348] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:30.150770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780eb8e3-7bc7-4f11-acc7-e26802297914 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:30.152295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-780eb8e3-7bc7-4f11-acc7-e26802297914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:30.208990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:30.254377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780eb8e3-7bc7-4f11-acc7-e26802297914 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:30.326325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33c26d19-e559-4248-b432-d266fa142ec8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:30.326926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c26d19-e559-4248-b432-d266fa142ec8 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:30.327198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c26d19-e559-4248-b432-d266fa142ec8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:30.327466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c26d19-e559-4248-b432-d266fa142ec8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:30.329073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-33c26d19-e559-4248-b432-d266fa142ec8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:30.434099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:30.438376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c26d19-e559-4248-b432-d266fa142ec8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:30.439812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33c26d19-e559-4248-b432-d266fa142ec8 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:30.458346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 669/1349] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:35.451412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b8a256-443a-47ad-be98-8beaadeabd30 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:35.451900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b8a256-443a-47ad-be98-8beaadeabd30 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:35.451985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b8a256-443a-47ad-be98-8beaadeabd30 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:35.453602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 681/1350] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:10:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:35.458634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f04d823-1afb-4593-8aeb-2f7f8f1e0e46 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:35.459077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f04d823-1afb-4593-8aeb-2f7f8f1e0e46 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:35.459249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f04d823-1afb-4593-8aeb-2f7f8f1e0e46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:35.591497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1f04d823-1afb-4593-8aeb-2f7f8f1e0e46 None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:35.592795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1f04d823-1afb-4593-8aeb-2f7f8f1e0e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:35.737780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 670/1351] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:10:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:39.820033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648628e-3e56-46e4-b57d-c428cb5bf07c None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:39.820528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648628e-3e56-46e4-b57d-c428cb5bf07c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:39.820528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648628e-3e56-46e4-b57d-c428cb5bf07c None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:39.822024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 682/1352] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:10:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:39.827028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03295655-c17a-4f99-8009-fb7241e7e645 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:39.827435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03295655-c17a-4f99-8009-fb7241e7e645 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:39.827640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03295655-c17a-4f99-8009-fb7241e7e645 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:39.866160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=59 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:39.883403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:39.907934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-03295655-c17a-4f99-8009-fb7241e7e645 None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:39.909146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-03295655-c17a-4f99-8009-fb7241e7e645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:40.020732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 671/1353] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:10:39 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:10:41.500398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a262c6-e77c-4887-b777-4cfc596ccfb8 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:41.500890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a262c6-e77c-4887-b777-4cfc596ccfb8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:41.501000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a262c6-e77c-4887-b777-4cfc596ccfb8 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:41.502316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 683/1354] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:10:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:41.508171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4970ecb2-48ad-4d3d-a946-9fb9879f6f83 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:41.508610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4970ecb2-48ad-4d3d-a946-9fb9879f6f83 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:41.508723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4970ecb2-48ad-4d3d-a946-9fb9879f6f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:41.586230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4970ecb2-48ad-4d3d-a946-9fb9879f6f83 None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:41.587350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4970ecb2-48ad-4d3d-a946-9fb9879f6f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:41.629291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:41.694979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 672/1355] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:10:41 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:45.582969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbce8c6-31cb-4f9f-9068-c770b9141bd7 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:45.584757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbce8c6-31cb-4f9f-9068-c770b9141bd7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:45.585307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbce8c6-31cb-4f9f-9068-c770b9141bd7 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:45.592634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e8864f-1daa-418e-a320-45291ca528aa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:45.593233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 684/1356] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:45 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:45.593658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e8864f-1daa-418e-a320-45291ca528aa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:45.594025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e8864f-1daa-418e-a320-45291ca528aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:45.664223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f2e8864f-1daa-418e-a320-45291ca528aa None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:45.665454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f2e8864f-1daa-418e-a320-45291ca528aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:45.773035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 673/1357] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:46.077859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d93de-053b-4636-bf17-03210393b883 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:46.078889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d93de-053b-4636-bf17-03210393b883 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:46.079703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d93de-053b-4636-bf17-03210393b883 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:46.083284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 685/1358] 10.222.0.26 () {54 vars in 961 bytes} [Sat Oct 1 04:10:46 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:46.092030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f777a0-53b4-4dc6-b7c2-d8d18c96d0ba None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:46.093180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f777a0-53b4-4dc6-b7c2-d8d18c96d0ba None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:46.093817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f777a0-53b4-4dc6-b7c2-d8d18c96d0ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:46.112068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:46.188125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-63f777a0-53b4-4dc6-b7c2-d8d18c96d0ba None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:46.189729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-63f777a0-53b4-4dc6-b7c2-d8d18c96d0ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:46.314131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 674/1359] 10.222.0.26 () {58 vars in 1058 bytes} [Sat Oct 1 04:10:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:46.322424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27d93de-053b-4636-bf17-03210393b883 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:46.323919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e27d93de-053b-4636-bf17-03210393b883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:46.477804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27d93de-053b-4636-bf17-03210393b883 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:46.561347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:46.609490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6118d7-2720-4a2f-95cd-a7cd64d9f1a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:46.610131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6118d7-2720-4a2f-95cd-a7cd64d9f1a7 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:46.610370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6118d7-2720-4a2f-95cd-a7cd64d9f1a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:46.610576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6118d7-2720-4a2f-95cd-a7cd64d9f1a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:46.612280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fe6118d7-2720-4a2f-95cd-a7cd64d9f1a7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:46.712907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6118d7-2720-4a2f-95cd-a7cd64d9f1a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:46.714542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6118d7-2720-4a2f-95cd-a7cd64d9f1a7 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:46.734279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 686/1360] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:49.167199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cbe1b2-a70a-483a-b680-a2fa8331426e None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:49.167876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cbe1b2-a70a-483a-b680-a2fa8331426e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:49.167876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cbe1b2-a70a-483a-b680-a2fa8331426e None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:49.173029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 675/1361] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:49 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:49.174323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1e606-8455-49f5-a75b-97b7643e6d40 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:49.174788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1e606-8455-49f5-a75b-97b7643e6d40 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:49.174788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1e606-8455-49f5-a75b-97b7643e6d40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:49.252404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5db1e606-8455-49f5-a75b-97b7643e6d40 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:49.253776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5db1e606-8455-49f5-a75b-97b7643e6d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:49.293967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:49.358475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 687/1362] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:49.655711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cbe1b2-a70a-483a-b680-a2fa8331426e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:49.657272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-54cbe1b2-a70a-483a-b680-a2fa8331426e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:49.741976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:49.845601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cbe1b2-a70a-483a-b680-a2fa8331426e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:49.982269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfab91f-cad9-4678-a937-825f7d34da45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:49.982882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfab91f-cad9-4678-a937-825f7d34da45 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:49.983167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfab91f-cad9-4678-a937-825f7d34da45 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:49.983443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfab91f-cad9-4678-a937-825f7d34da45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:49.985094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4cfab91f-cad9-4678-a937-825f7d34da45 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:50.008406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:50.080502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfab91f-cad9-4678-a937-825f7d34da45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:50.082155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfab91f-cad9-4678-a937-825f7d34da45 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:50.101814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 676/1363] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:52.671068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05d0aaf-de8b-4d4b-993c-75c7a7122661 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:52.672054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05d0aaf-de8b-4d4b-993c-75c7a7122661 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:52.672483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05d0aaf-de8b-4d4b-993c-75c7a7122661 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:52.674043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 688/1364] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:52.680601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3908d-7d58-4e0d-90f4-25b1b4d47d35 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:52.681514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3908d-7d58-4e0d-90f4-25b1b4d47d35 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:52.681971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3908d-7d58-4e0d-90f4-25b1b4d47d35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:52.764896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e1e3908d-7d58-4e0d-90f4-25b1b4d47d35 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:52.766387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e1e3908d-7d58-4e0d-90f4-25b1b4d47d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:52.879538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:52.893454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 677/1365] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:53.194027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb437c8-7925-4f0d-8d4b-f0584bc64395 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:53.194425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb437c8-7925-4f0d-8d4b-f0584bc64395 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:53.194716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb437c8-7925-4f0d-8d4b-f0584bc64395 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:53.196430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 689/1366] 10.222.0.26 () {54 vars in 961 bytes} [Sat Oct 1 04:10:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:53.201686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d61454-3ca5-461f-ba4a-7a7c87b5e526 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:53.202542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d61454-3ca5-461f-ba4a-7a7c87b5e526 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:53.202896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d61454-3ca5-461f-ba4a-7a7c87b5e526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:53.313123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e2d61454-3ca5-461f-ba4a-7a7c87b5e526 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:53.314326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e2d61454-3ca5-461f-ba4a-7a7c87b5e526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:53.447883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 678/1367] 10.222.0.26 () {58 vars in 1058 bytes} [Sat Oct 1 04:10:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:53.456064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb437c8-7925-4f0d-8d4b-f0584bc64395 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:53.457690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5bb437c8-7925-4f0d-8d4b-f0584bc64395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:53.637380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb437c8-7925-4f0d-8d4b-f0584bc64395 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:53.659895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:53.761022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb511b5-c68a-46cb-a100-c7d6badfd409 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:53.772720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb511b5-c68a-46cb-a100-c7d6badfd409 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:53.772720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb511b5-c68a-46cb-a100-c7d6badfd409 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:53.772720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb511b5-c68a-46cb-a100-c7d6badfd409 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:53.772720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9eb511b5-c68a-46cb-a100-c7d6badfd409 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:53.871077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb511b5-c68a-46cb-a100-c7d6badfd409 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:53.873278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb511b5-c68a-46cb-a100-c7d6badfd409 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:53.896269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 690/1368] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:54.778960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd717b-1a97-4ddd-b461-b6e29fe83187 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:54.779499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd717b-1a97-4ddd-b461-b6e29fe83187 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:54.780117 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd717b-1a97-4ddd-b461-b6e29fe83187 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:54.782013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 679/1369] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:54.789761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41735ccd-864a-4f48-aa8e-5797bf961d53 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:54.790349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41735ccd-864a-4f48-aa8e-5797bf961d53 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:54.790349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41735ccd-864a-4f48-aa8e-5797bf961d53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:54.835051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:54.883328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-41735ccd-864a-4f48-aa8e-5797bf961d53 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:54.884475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-41735ccd-864a-4f48-aa8e-5797bf961d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:54.997225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 691/1370] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:55.512982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e630950-e1c6-4a99-97e4-9e01e36fbb92 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:55.513415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e630950-e1c6-4a99-97e4-9e01e36fbb92 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:55.513834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e630950-e1c6-4a99-97e4-9e01e36fbb92 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:55.518869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 680/1371] 10.222.0.26 () {54 vars in 963 bytes} [Sat Oct 1 04:10:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:55.523629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccbfa32-b446-487e-a8bd-898a52872024 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:55.524058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccbfa32-b446-487e-a8bd-898a52872024 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:55.524460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccbfa32-b446-487e-a8bd-898a52872024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:55.637082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-6ccbfa32-b446-487e-a8bd-898a52872024 None None] MFA Rules not processed for user `52b5d64c999a4f16afec678df3025529`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:55.638764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6ccbfa32-b446-487e-a8bd-898a52872024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:55.707110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:55.752111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 692/1372] 10.222.0.26 () {58 vars in 1060 bytes} [Sat Oct 1 04:10:55 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:55.763289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e630950-e1c6-4a99-97e4-9e01e36fbb92 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:55.764818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8e630950-e1c6-4a99-97e4-9e01e36fbb92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:55.817449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:55.873573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e630950-e1c6-4a99-97e4-9e01e36fbb92 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:55.955141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157468ee-fda5-4186-8152-4f826f332555 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:55.955752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157468ee-fda5-4186-8152-4f826f332555 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:55.956031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157468ee-fda5-4186-8152-4f826f332555 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:55.956303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157468ee-fda5-4186-8152-4f826f332555 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:55.957930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-157468ee-fda5-4186-8152-4f826f332555 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:56.062953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:56.079142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157468ee-fda5-4186-8152-4f826f332555 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:56.080599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157468ee-fda5-4186-8152-4f826f332555 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:56.100599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 681/1373] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:56.170329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e58845-5b00-40a6-8853-dea5b6fafd38 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:56.170900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e58845-5b00-40a6-8853-dea5b6fafd38 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:56.170999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e58845-5b00-40a6-8853-dea5b6fafd38 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:56.172463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 693/1374] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:56.178447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f498b9-0938-4464-89e3-3a1b943a9ef2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:56.178740 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f498b9-0938-4464-89e3-3a1b943a9ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:56.179015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f498b9-0938-4464-89e3-3a1b943a9ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:56.245413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c3f498b9-0938-4464-89e3-3a1b943a9ef2 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:56.246544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c3f498b9-0938-4464-89e3-3a1b943a9ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:56.344861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 682/1375] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:56.656806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e58845-5b00-40a6-8853-dea5b6fafd38 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:56.658565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-58e58845-5b00-40a6-8853-dea5b6fafd38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:56.815693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e58845-5b00-40a6-8853-dea5b6fafd38 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:10:56.870809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:56.933596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8723afa-94be-4f05-98cc-1a49772c78f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:10:56.934195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8723afa-94be-4f05-98cc-1a49772c78f0 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:56.934470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8723afa-94be-4f05-98cc-1a49772c78f0 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:56.934737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8723afa-94be-4f05-98cc-1a49772c78f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:56.936301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c8723afa-94be-4f05-98cc-1a49772c78f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:57.031671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8723afa-94be-4f05-98cc-1a49772c78f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:10:57.033381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8723afa-94be-4f05-98cc-1a49772c78f0 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:10:57.054339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 694/1376] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:10:59.242321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838da24d-93fc-4a74-99b1-f34461abfa56 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:59.242831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838da24d-93fc-4a74-99b1-f34461abfa56 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:59.242925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838da24d-93fc-4a74-99b1-f34461abfa56 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:59.244215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 683/1377] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:10:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:10:59.249571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d58f93f-c7f5-4406-aed6-ac34714e3b69 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:10:59.249965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d58f93f-c7f5-4406-aed6-ac34714e3b69 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:10:59.250072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d58f93f-c7f5-4406-aed6-ac34714e3b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:10:59.322405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9d58f93f-c7f5-4406-aed6-ac34714e3b69 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:10:59.323594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9d58f93f-c7f5-4406-aed6-ac34714e3b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:59.330777 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:59.431604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 695/1378] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:10:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:10:59.739478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838da24d-93fc-4a74-99b1-f34461abfa56 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:10:59.743197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-838da24d-93fc-4a74-99b1-f34461abfa56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:10:59.813062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:10:59.905443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838da24d-93fc-4a74-99b1-f34461abfa56 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:00.039727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df55de6f-6d24-4a04-9247-f20f6b761b17 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:00.040632 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df55de6f-6d24-4a04-9247-f20f6b761b17 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:00.041169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df55de6f-6d24-4a04-9247-f20f6b761b17 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:00.041657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df55de6f-6d24-4a04-9247-f20f6b761b17 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:00.043424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-df55de6f-6d24-4a04-9247-f20f6b761b17 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:00.064524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:00.157384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df55de6f-6d24-4a04-9247-f20f6b761b17 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:00.159143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df55de6f-6d24-4a04-9247-f20f6b761b17 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:00.178415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 684/1379] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:10:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:02.450284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f6d76-021b-48ec-a46b-d6b480639f3b None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:02.450823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f6d76-021b-48ec-a46b-d6b480639f3b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:02.450823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3f6d76-021b-48ec-a46b-d6b480639f3b None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:02.452353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 696/1380] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:02.457523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd13c6a-f9e7-4bcd-baac-e535ef48a6a5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:02.457918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd13c6a-f9e7-4bcd-baac-e535ef48a6a5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:02.458136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd13c6a-f9e7-4bcd-baac-e535ef48a6a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:02.539220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3dd13c6a-f9e7-4bcd-baac-e535ef48a6a5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:02.540674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3dd13c6a-f9e7-4bcd-baac-e535ef48a6a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:02.648788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:02.657269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 685/1381] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:02.948417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f6d76-021b-48ec-a46b-d6b480639f3b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:02.950136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7d3f6d76-021b-48ec-a46b-d6b480639f3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:03.111142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:03.129092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3f6d76-021b-48ec-a46b-d6b480639f3b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:03.258635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847cc8ee-b155-4a39-8d87-3b7c1443eb9b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:03.259460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847cc8ee-b155-4a39-8d87-3b7c1443eb9b service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:03.259937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847cc8ee-b155-4a39-8d87-3b7c1443eb9b service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:03.260401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847cc8ee-b155-4a39-8d87-3b7c1443eb9b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:03.262127 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-847cc8ee-b155-4a39-8d87-3b7c1443eb9b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:03.370681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847cc8ee-b155-4a39-8d87-3b7c1443eb9b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:03.372202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847cc8ee-b155-4a39-8d87-3b7c1443eb9b service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:03.383056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:03.392337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 697/1382] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:05.652536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5f6ba1-9487-49d4-85a2-0550187a5dce None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:05.653603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5f6ba1-9487-49d4-85a2-0550187a5dce None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:05.653603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5f6ba1-9487-49d4-85a2-0550187a5dce None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:05.654271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 686/1383] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:05.659639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef0912-1f25-43bb-a906-cbf75a03e900 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:05.660213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef0912-1f25-43bb-a906-cbf75a03e900 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:05.660375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef0912-1f25-43bb-a906-cbf75a03e900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:05.735411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-02ef0912-1f25-43bb-a906-cbf75a03e900 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:05.736678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-02ef0912-1f25-43bb-a906-cbf75a03e900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:05.840965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 698/1384] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:06.131366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5f6ba1-9487-49d4-85a2-0550187a5dce None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:06.132895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3c5f6ba1-9487-49d4-85a2-0550187a5dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:06.282587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5f6ba1-9487-49d4-85a2-0550187a5dce None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:06.350409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:06.399194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49500b42-207c-4b31-af05-4d42c9234f17 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:06.399729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500b42-207c-4b31-af05-4d42c9234f17 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:06.400007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500b42-207c-4b31-af05-4d42c9234f17 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:06.400276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500b42-207c-4b31-af05-4d42c9234f17 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:06.402162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-49500b42-207c-4b31-af05-4d42c9234f17 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:06.493616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49500b42-207c-4b31-af05-4d42c9234f17 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:06.495208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49500b42-207c-4b31-af05-4d42c9234f17 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:06.514425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 687/1385] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:09.361861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117990a4-3311-42ff-9dd2-bfcdf3d257ee None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:09.362403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117990a4-3311-42ff-9dd2-bfcdf3d257ee None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:09.362403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117990a4-3311-42ff-9dd2-bfcdf3d257ee None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:09.363880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 699/1386] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:09.370428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a51b28-d76f-46c1-abd6-bb2a78f816ae None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:09.373828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a51b28-d76f-46c1-abd6-bb2a78f816ae None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:09.373828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a51b28-d76f-46c1-abd6-bb2a78f816ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:09.438412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-92a51b28-d76f-46c1-abd6-bb2a78f816ae None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:09.439621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-92a51b28-d76f-46c1-abd6-bb2a78f816ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:09.474193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:09.542897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 688/1387] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:09.830926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117990a4-3311-42ff-9dd2-bfcdf3d257ee None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:09.832495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-117990a4-3311-42ff-9dd2-bfcdf3d257ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:09.910753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:09.982917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117990a4-3311-42ff-9dd2-bfcdf3d257ee None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:10.098705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e148a3a9-9e3c-4407-98c9-739d7f5517f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:10.099869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148a3a9-9e3c-4407-98c9-739d7f5517f9 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:10.100176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148a3a9-9e3c-4407-98c9-739d7f5517f9 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:10.100448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148a3a9-9e3c-4407-98c9-739d7f5517f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:10.102097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e148a3a9-9e3c-4407-98c9-739d7f5517f9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:10.142082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:10.197186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e148a3a9-9e3c-4407-98c9-739d7f5517f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:10.198789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e148a3a9-9e3c-4407-98c9-739d7f5517f9 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:10.217466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 700/1388] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:11.437803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a1a4d-c335-4b43-9e2c-f24cf0eecc0a None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:11.438338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a1a4d-c335-4b43-9e2c-f24cf0eecc0a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:11.438338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09a1a4d-c335-4b43-9e2c-f24cf0eecc0a None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:11.439711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 689/1389] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:11.446066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8c60d-d8e8-44f0-850e-d8c5afe540d8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:11.446483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8c60d-d8e8-44f0-850e-d8c5afe540d8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:11.446599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8c60d-d8e8-44f0-850e-d8c5afe540d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:11.514550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-abc8c60d-d8e8-44f0-850e-d8c5afe540d8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:11.515681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-abc8c60d-d8e8-44f0-850e-d8c5afe540d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:11.616357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 701/1390] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:11.925515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d017b715-c8e2-4053-b5d0-82375ba65308 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:11.925944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d017b715-c8e2-4053-b5d0-82375ba65308 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:11.926230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d017b715-c8e2-4053-b5d0-82375ba65308 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:11.928340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 690/1391] 10.222.0.26 () {54 vars in 962 bytes} [Sat Oct 1 04:11:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:11.933608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7866c2c4-e95c-4221-bb8e-25336792e9e6 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:11.934066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7866c2c4-e95c-4221-bb8e-25336792e9e6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:11.934427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7866c2c4-e95c-4221-bb8e-25336792e9e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:11.951660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:12.019367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7866c2c4-e95c-4221-bb8e-25336792e9e6 None None] MFA Rules not processed for user `170fe0394f3d4472a1380481cfe2cc28`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:12.021290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7866c2c4-e95c-4221-bb8e-25336792e9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:12.105736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 702/1392] 10.222.0.26 () {58 vars in 1059 bytes} [Sat Oct 1 04:11:11 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:11:12.115964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d017b715-c8e2-4053-b5d0-82375ba65308 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:12.117876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d017b715-c8e2-4053-b5d0-82375ba65308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:12.230142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d017b715-c8e2-4053-b5d0-82375ba65308 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:12.289851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:12.307887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:12.308515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:12.308788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:12.309097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:12.310786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:12.420784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abff429-1c44-4f75-b094-52e82af11450 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:12.421130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abff429-1c44-4f75-b094-52e82af11450 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:12.421409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abff429-1c44-4f75-b094-52e82af11450 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:12.422778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 703/1393] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:11:12.424459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:12.426005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:12.428839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c30cbd-0dc6-43ca-b476-a63643178a67 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:12.429176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c30cbd-0dc6-43ca-b476-a63643178a67 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:12.429455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c30cbd-0dc6-43ca-b476-a63643178a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:12.446304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 691/1394] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:12.510266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-49c30cbd-0dc6-43ca-b476-a63643178a67 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:12.511394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-49c30cbd-0dc6-43ca-b476-a63643178a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:12.589336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:12.632020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 704/1395] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:11:12.931516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:12.932694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:13.026745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:13.036132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fc7406-4adc-40fa-8290-9342fb265a75 service glance] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:13.161179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f821c50-8f21-48de-8f12-37fff3d5c09a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:13.162325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f821c50-8f21-48de-8f12-37fff3d5c09a service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:13.162631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f821c50-8f21-48de-8f12-37fff3d5c09a service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:13.162872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f821c50-8f21-48de-8f12-37fff3d5c09a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:13.164499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9f821c50-8f21-48de-8f12-37fff3d5c09a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:13.212484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa72ee8-77b6-4141-85cf-e0adc650b7c2 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:13.212848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa72ee8-77b6-4141-85cf-e0adc650b7c2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:13.213217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa72ee8-77b6-4141-85cf-e0adc650b7c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:13.263200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f821c50-8f21-48de-8f12-37fff3d5c09a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:13.264821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f821c50-8f21-48de-8f12-37fff3d5c09a service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:13.280484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-faa72ee8-77b6-4141-85cf-e0adc650b7c2 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:13.280748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:13.282292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-faa72ee8-77b6-4141-85cf-e0adc650b7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:13.283957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 692/1396] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:13.377872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 705/1397] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:11:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:11:13.424780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadbfb50-1877-49ed-992f-406744b13421 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:13.425103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadbfb50-1877-49ed-992f-406744b13421 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:13.425382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadbfb50-1877-49ed-992f-406744b13421 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:13.426796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 693/1398] 10.222.0.26 () {54 vars in 943 bytes} [Sat Oct 1 04:11:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:13.435436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a728103-244f-4215-9217-52155b6df339 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:13.435731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a728103-244f-4215-9217-52155b6df339 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:13.436049 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a728103-244f-4215-9217-52155b6df339 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:13.504972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:13.526706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7a728103-244f-4215-9217-52155b6df339 None None] MFA Rules not processed for user `3d69adfda7334567b03627a1d839f4f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:13.529101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7a728103-244f-4215-9217-52155b6df339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:13.637437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 706/1399] 10.222.0.26 () {58 vars in 1040 bytes} [Sat Oct 1 04:11:13 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:13.647406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadbfb50-1877-49ed-992f-406744b13421 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:13.648974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cadbfb50-1877-49ed-992f-406744b13421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:13.784382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadbfb50-1877-49ed-992f-406744b13421 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:13.871000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:13.890493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56edc18a-d4dc-4ccd-9df9-166c02d93aac service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:13.891094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56edc18a-d4dc-4ccd-9df9-166c02d93aac service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:13.891368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56edc18a-d4dc-4ccd-9df9-166c02d93aac service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:13.891634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56edc18a-d4dc-4ccd-9df9-166c02d93aac service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:13.893262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-56edc18a-d4dc-4ccd-9df9-166c02d93aac service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:13.976316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56edc18a-d4dc-4ccd-9df9-166c02d93aac service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:13.977997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56edc18a-d4dc-4ccd-9df9-166c02d93aac service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:13.986054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 694/1400] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:11:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:11:15.496895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed3638-b73a-45e9-af5e-c919e9d3e24c None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:15.497475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed3638-b73a-45e9-af5e-c919e9d3e24c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:15.497475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed3638-b73a-45e9-af5e-c919e9d3e24c None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:15.498958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 707/1401] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:15.507292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86671bfa-edb8-4f94-b276-c2897dd5a28b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:15.511788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86671bfa-edb8-4f94-b276-c2897dd5a28b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:15.512246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86671bfa-edb8-4f94-b276-c2897dd5a28b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:15.604190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-86671bfa-edb8-4f94-b276-c2897dd5a28b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:15.605340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-86671bfa-edb8-4f94-b276-c2897dd5a28b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:15.697153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:15.731672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 695/1402] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:16.024080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ed3638-b73a-45e9-af5e-c919e9d3e24c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:16.025644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-07ed3638-b73a-45e9-af5e-c919e9d3e24c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:16.167646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:16.224375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ed3638-b73a-45e9-af5e-c919e9d3e24c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:16.346958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6278bd3c-ddc9-4afc-b0a5-c72f00dcaaa6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:16.347504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6278bd3c-ddc9-4afc-b0a5-c72f00dcaaa6 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:16.347780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6278bd3c-ddc9-4afc-b0a5-c72f00dcaaa6 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:16.348048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6278bd3c-ddc9-4afc-b0a5-c72f00dcaaa6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:16.350388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6278bd3c-ddc9-4afc-b0a5-c72f00dcaaa6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:16.430911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:16.453301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6278bd3c-ddc9-4afc-b0a5-c72f00dcaaa6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:16.454895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6278bd3c-ddc9-4afc-b0a5-c72f00dcaaa6 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:16.474153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 708/1403] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:18.967606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2a9350-9864-4de2-944f-22d68b158a5f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:18.968200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2a9350-9864-4de2-944f-22d68b158a5f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:18.968200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2a9350-9864-4de2-944f-22d68b158a5f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:18.973383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 696/1404] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:18 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:18.975473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d8afcc-717e-4634-a85c-76bfe30ee615 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:18.975914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d8afcc-717e-4634-a85c-76bfe30ee615 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:18.976066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d8afcc-717e-4634-a85c-76bfe30ee615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:19.047488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-76d8afcc-717e-4634-a85c-76bfe30ee615 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:19.048723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-76d8afcc-717e-4634-a85c-76bfe30ee615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:19.157859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 709/1405] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:19.443186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2a9350-9864-4de2-944f-22d68b158a5f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:19.444689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bd2a9350-9864-4de2-944f-22d68b158a5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:19.612530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2a9350-9864-4de2-944f-22d68b158a5f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:19.676576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:19.870341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e5701f-0701-49e6-ad2f-e3e2a9c4d2f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:19.871132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e5701f-0701-49e6-ad2f-e3e2a9c4d2f5 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:19.871407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e5701f-0701-49e6-ad2f-e3e2a9c4d2f5 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:19.871674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e5701f-0701-49e6-ad2f-e3e2a9c4d2f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:19.874573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b6e5701f-0701-49e6-ad2f-e3e2a9c4d2f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:19.980522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e5701f-0701-49e6-ad2f-e3e2a9c4d2f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:19.982868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e5701f-0701-49e6-ad2f-e3e2a9c4d2f5 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:20.001530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 697/1406] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:11:22.242932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3f14d-ef0d-4b4c-af1b-de10e4b9bf3d None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:22.243511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3f14d-ef0d-4b4c-af1b-de10e4b9bf3d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:22.243511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a3f14d-ef0d-4b4c-af1b-de10e4b9bf3d None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:22.244992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 710/1407] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:11:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:22.250343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0b02a3-73e0-425d-a28f-66dd9f707d4b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:22.250822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0b02a3-73e0-425d-a28f-66dd9f707d4b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:22.250921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0b02a3-73e0-425d-a28f-66dd9f707d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:22.326099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1c0b02a3-73e0-425d-a28f-66dd9f707d4b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:22.327324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1c0b02a3-73e0-425d-a28f-66dd9f707d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:22.342171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:22.433411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 698/1408] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:11:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:22.730577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a3f14d-ef0d-4b4c-af1b-de10e4b9bf3d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:22.732144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-84a3f14d-ef0d-4b4c-af1b-de10e4b9bf3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:22.791444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:22.904630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a3f14d-ef0d-4b4c-af1b-de10e4b9bf3d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:23.043104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa1ece0-4dc3-480c-94ba-de79f919f745 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:23.043628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa1ece0-4dc3-480c-94ba-de79f919f745 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:23.043908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa1ece0-4dc3-480c-94ba-de79f919f745 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:23.044190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa1ece0-4dc3-480c-94ba-de79f919f745 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:23.045795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5aa1ece0-4dc3-480c-94ba-de79f919f745 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:23.056804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:23.154767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa1ece0-4dc3-480c-94ba-de79f919f745 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:23.156416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa1ece0-4dc3-480c-94ba-de79f919f745 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:23.193128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 711/1409] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:32.432009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=37 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:33.174460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=41 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:37.161323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bad218-b2a4-452a-a7ef-44d7e62faace None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:37.161898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bad218-b2a4-452a-a7ef-44d7e62faace None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:37.161989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bad218-b2a4-452a-a7ef-44d7e62faace None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:37.163355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 699/1410] 10.222.0.26 () {54 vars in 918 bytes} [Sat Oct 1 04:11:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:37.168364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606b6c15-b14f-42bc-8305-ea04b42e33a2 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:37.168775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606b6c15-b14f-42bc-8305-ea04b42e33a2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:37.168918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606b6c15-b14f-42bc-8305-ea04b42e33a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:37.243288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-606b6c15-b14f-42bc-8305-ea04b42e33a2 None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:37.244493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-606b6c15-b14f-42bc-8305-ea04b42e33a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:37.347795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 712/1411] 10.222.0.26 () {58 vars in 1015 bytes} [Sat Oct 1 04:11:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:37.362335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302e69ee-6650-4411-afa2-23aa06235352 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:37.362620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302e69ee-6650-4411-afa2-23aa06235352 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:37.362895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302e69ee-6650-4411-afa2-23aa06235352 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:37.364828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 700/1412] 10.222.0.26 () {54 vars in 918 bytes} [Sat Oct 1 04:11:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:37.369569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44cd553-094e-41bb-b0b8-505776ec01d3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:37.369850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44cd553-094e-41bb-b0b8-505776ec01d3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:37.370129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44cd553-094e-41bb-b0b8-505776ec01d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:37.441643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:37.445351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a44cd553-094e-41bb-b0b8-505776ec01d3 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:37.447055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a44cd553-094e-41bb-b0b8-505776ec01d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:37.561231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 713/1413] 10.222.0.26 () {58 vars in 1015 bytes} [Sat Oct 1 04:11:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:11:40.137579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265a5d68-1058-4c5d-bec0-b3d9490433d7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:40.138147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265a5d68-1058-4c5d-bec0-b3d9490433d7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:40.138147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265a5d68-1058-4c5d-bec0-b3d9490433d7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:40.139471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 701/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:11:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:40.144890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c6378b-b69f-42f3-b186-14809c0105c4 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:40.145347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c6378b-b69f-42f3-b186-14809c0105c4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:40.145899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c6378b-b69f-42f3-b186-14809c0105c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:40.213882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e4c6378b-b69f-42f3-b186-14809c0105c4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:40.215091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e4c6378b-b69f-42f3-b186-14809c0105c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:40.269232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:40.309687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 714/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:11:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:40.601453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265a5d68-1058-4c5d-bec0-b3d9490433d7 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:40.603415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-265a5d68-1058-4c5d-bec0-b3d9490433d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:40.748662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265a5d68-1058-4c5d-bec0-b3d9490433d7 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:40.819854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:40.858773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f67f4f-41b8-4374-9e21-f8b490623581 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:40.859379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f67f4f-41b8-4374-9e21-f8b490623581 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:40.859656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f67f4f-41b8-4374-9e21-f8b490623581 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:40.859925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f67f4f-41b8-4374-9e21-f8b490623581 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:40.861525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f6f67f4f-41b8-4374-9e21-f8b490623581 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:40.949752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f67f4f-41b8-4374-9e21-f8b490623581 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:40.951381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f67f4f-41b8-4374-9e21-f8b490623581 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:40.969427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 702/1416] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:11:42.106404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a461a-400d-443d-831e-ca43409a8bc0 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:42.106972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a461a-400d-443d-831e-ca43409a8bc0 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:42.106972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a461a-400d-443d-831e-ca43409a8bc0 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:42.108318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 715/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:42.115648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737fe853-5d25-4cd9-a706-25fa57fcff95 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:42.116095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737fe853-5d25-4cd9-a706-25fa57fcff95 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:42.116190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737fe853-5d25-4cd9-a706-25fa57fcff95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:42.190902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-737fe853-5d25-4cd9-a706-25fa57fcff95 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:42.192140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-737fe853-5d25-4cd9-a706-25fa57fcff95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:42.218384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:42.288084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 703/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:42.443791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13a461a-400d-443d-831e-ca43409a8bc0 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:42.445340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d13a461a-400d-443d-831e-ca43409a8bc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:42.588878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13a461a-400d-443d-831e-ca43409a8bc0 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:42.628178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:42.705154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d490b28a-e731-474b-859b-ce842c2b6160 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:42.705692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490b28a-e731-474b-859b-ce842c2b6160 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:42.705971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490b28a-e731-474b-859b-ce842c2b6160 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:42.706243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490b28a-e731-474b-859b-ce842c2b6160 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:42.708296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d490b28a-e731-474b-859b-ce842c2b6160 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:42.798024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d490b28a-e731-474b-859b-ce842c2b6160 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:42.799599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d490b28a-e731-474b-859b-ce842c2b6160 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:42.818834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 716/1419] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:43.975840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4c430-8664-4ad1-b0b9-b91ef4bf9ccb None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:43.976385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4c430-8664-4ad1-b0b9-b91ef4bf9ccb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:43.976385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4c430-8664-4ad1-b0b9-b91ef4bf9ccb None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:43.977812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 704/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:43.983844 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d599acf8-c9f6-4f3f-a0d8-cc12cddb6c51 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:43.984286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d599acf8-c9f6-4f3f-a0d8-cc12cddb6c51 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:43.984420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d599acf8-c9f6-4f3f-a0d8-cc12cddb6c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:44.155147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:44.178170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d599acf8-c9f6-4f3f-a0d8-cc12cddb6c51 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:44.179387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d599acf8-c9f6-4f3f-a0d8-cc12cddb6c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:44.273295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 717/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:44.425373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4c430-8664-4ad1-b0b9-b91ef4bf9ccb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:44.427536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7bc4c430-8664-4ad1-b0b9-b91ef4bf9ccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:44.571808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc4c430-8664-4ad1-b0b9-b91ef4bf9ccb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:44.583597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:44.681757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeca66e-02b1-480b-ba8d-48b18f75959f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:44.682387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeca66e-02b1-480b-ba8d-48b18f75959f service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:44.682661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeca66e-02b1-480b-ba8d-48b18f75959f service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:44.682928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeca66e-02b1-480b-ba8d-48b18f75959f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:44.684500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3aeca66e-02b1-480b-ba8d-48b18f75959f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:44.772193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeca66e-02b1-480b-ba8d-48b18f75959f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:44.773798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeca66e-02b1-480b-ba8d-48b18f75959f service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:44.793498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 705/1422] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:45.995498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a7019-a730-4612-8f88-d22914c2b3c7 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:45.996079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a7019-a730-4612-8f88-d22914c2b3c7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:45.996079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a7019-a730-4612-8f88-d22914c2b3c7 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:45.997580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 718/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:46.004025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e765ae-e97f-447a-932a-f606986a25e9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:46.005063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e765ae-e97f-447a-932a-f606986a25e9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:46.005168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e765ae-e97f-447a-932a-f606986a25e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:46.025347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:46.081986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-93e765ae-e97f-447a-932a-f606986a25e9 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:46.083851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-93e765ae-e97f-447a-932a-f606986a25e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:46.184445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 706/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:46.336289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a7019-a730-4612-8f88-d22914c2b3c7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:46.337722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3a0a7019-a730-4612-8f88-d22914c2b3c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:46.450212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:46.484757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a7019-a730-4612-8f88-d22914c2b3c7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:46.594925 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8424ec1a-c869-4fa4-a348-27dcf6549d93 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:46.595472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8424ec1a-c869-4fa4-a348-27dcf6549d93 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:46.595748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8424ec1a-c869-4fa4-a348-27dcf6549d93 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:46.596012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8424ec1a-c869-4fa4-a348-27dcf6549d93 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:46.597635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8424ec1a-c869-4fa4-a348-27dcf6549d93 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:46.673564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:46.688161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8424ec1a-c869-4fa4-a348-27dcf6549d93 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:46.689793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8424ec1a-c869-4fa4-a348-27dcf6549d93 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:46.707710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 719/1425] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:47.827154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7803be2c-0643-4491-a231-b8b084b8aeca None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:47.827761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7803be2c-0643-4491-a231-b8b084b8aeca None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:47.827972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7803be2c-0643-4491-a231-b8b084b8aeca None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:47.829955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 707/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:47.836317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b391cd8-c62d-493a-85d0-2a30365aa62c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:47.836808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b391cd8-c62d-493a-85d0-2a30365aa62c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:47.837022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b391cd8-c62d-493a-85d0-2a30365aa62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:47.914310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2b391cd8-c62d-493a-85d0-2a30365aa62c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:47.915431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2b391cd8-c62d-493a-85d0-2a30365aa62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:48.010008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 720/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:48.161821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7803be2c-0643-4491-a231-b8b084b8aeca None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:48.163312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7803be2c-0643-4491-a231-b8b084b8aeca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:48.246248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:48.306551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7803be2c-0643-4491-a231-b8b084b8aeca None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:48.416326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a305c498-f331-440b-ac87-fc375d344487 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:48.416956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a305c498-f331-440b-ac87-fc375d344487 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:48.417239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a305c498-f331-440b-ac87-fc375d344487 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:48.417510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a305c498-f331-440b-ac87-fc375d344487 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:48.419092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a305c498-f331-440b-ac87-fc375d344487 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:48.466966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:48.507778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a305c498-f331-440b-ac87-fc375d344487 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:48.509339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a305c498-f331-440b-ac87-fc375d344487 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:48.527782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 708/1428] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:49.588830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28dbe6c-d3d3-44ff-b872-475e8ddbddc8 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:49.589393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28dbe6c-d3d3-44ff-b872-475e8ddbddc8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:49.589393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28dbe6c-d3d3-44ff-b872-475e8ddbddc8 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:49.590737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 721/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:49.597060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40270037-83cc-4cbd-8c1b-b9635478c66a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:49.597506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40270037-83cc-4cbd-8c1b-b9635478c66a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:49.597613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40270037-83cc-4cbd-8c1b-b9635478c66a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:49.672127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-40270037-83cc-4cbd-8c1b-b9635478c66a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:49.673366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-40270037-83cc-4cbd-8c1b-b9635478c66a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:49.768895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 709/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:49.919223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28dbe6c-d3d3-44ff-b872-475e8ddbddc8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:49.920707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f28dbe6c-d3d3-44ff-b872-475e8ddbddc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:49.969731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:50.064699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28dbe6c-d3d3-44ff-b872-475e8ddbddc8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:50.177370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd375235-8e36-4e9a-8394-40d4e0c3e305 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:50.177966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd375235-8e36-4e9a-8394-40d4e0c3e305 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:50.178243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd375235-8e36-4e9a-8394-40d4e0c3e305 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:50.178511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd375235-8e36-4e9a-8394-40d4e0c3e305 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:50.180089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-bd375235-8e36-4e9a-8394-40d4e0c3e305 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:50.186919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:50.267790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd375235-8e36-4e9a-8394-40d4e0c3e305 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:50.269322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd375235-8e36-4e9a-8394-40d4e0c3e305 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:50.288214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 722/1431] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:51.350987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ac368-a910-46c0-8f64-9e58f4d4ca9c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:51.350987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ac368-a910-46c0-8f64-9e58f4d4ca9c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:51.351451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ac368-a910-46c0-8f64-9e58f4d4ca9c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:51.352366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 710/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:51.358782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1788caea-a6f2-41bf-b8d0-58a480d4fea8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:51.359201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1788caea-a6f2-41bf-b8d0-58a480d4fea8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:51.359319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1788caea-a6f2-41bf-b8d0-58a480d4fea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:51.445572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-1788caea-a6f2-41bf-b8d0-58a480d4fea8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:51.446773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1788caea-a6f2-41bf-b8d0-58a480d4fea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:51.512752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:51.542626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 723/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:51.691625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ac368-a910-46c0-8f64-9e58f4d4ca9c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:51.693470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-032ac368-a910-46c0-8f64-9e58f4d4ca9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:51.712556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:51.837933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ac368-a910-46c0-8f64-9e58f4d4ca9c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:51.923090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:51.949876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a292fe9a-2fa8-438a-8b83-78cb0cab56d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:51.950484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292fe9a-2fa8-438a-8b83-78cb0cab56d1 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:51.950760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292fe9a-2fa8-438a-8b83-78cb0cab56d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:51.951027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292fe9a-2fa8-438a-8b83-78cb0cab56d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:51.952620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a292fe9a-2fa8-438a-8b83-78cb0cab56d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:52.044185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a292fe9a-2fa8-438a-8b83-78cb0cab56d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:52.045821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a292fe9a-2fa8-438a-8b83-78cb0cab56d1 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:52.065244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 711/1434] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:53.116210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2621b61e-730d-4007-89f6-fe301dc91d9e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:53.116789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2621b61e-730d-4007-89f6-fe301dc91d9e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:53.116789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2621b61e-730d-4007-89f6-fe301dc91d9e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:53.118432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 724/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:53.125280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b194c3e-7a22-4621-8fb0-7e94c10f3feb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:53.125773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b194c3e-7a22-4621-8fb0-7e94c10f3feb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:53.125884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b194c3e-7a22-4621-8fb0-7e94c10f3feb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:53.201371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8b194c3e-7a22-4621-8fb0-7e94c10f3feb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:53.202591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8b194c3e-7a22-4621-8fb0-7e94c10f3feb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:53.244448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:53.298303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 712/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:53.447431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2621b61e-730d-4007-89f6-fe301dc91d9e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:53.448987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2621b61e-730d-4007-89f6-fe301dc91d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:53.591326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2621b61e-730d-4007-89f6-fe301dc91d9e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:53.643633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:53.701584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c2b877-defd-4bcf-83fe-a43e90e0658d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:53.702102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2b877-defd-4bcf-83fe-a43e90e0658d service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:53.702379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2b877-defd-4bcf-83fe-a43e90e0658d service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:53.702642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2b877-defd-4bcf-83fe-a43e90e0658d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:53.704201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a0c2b877-defd-4bcf-83fe-a43e90e0658d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:53.799524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c2b877-defd-4bcf-83fe-a43e90e0658d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:53.801117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c2b877-defd-4bcf-83fe-a43e90e0658d service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:53.818527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 725/1437] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:54.883224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1f548c-6e14-44b6-89da-9e1fed3c5a8c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:54.883758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1f548c-6e14-44b6-89da-9e1fed3c5a8c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:54.883758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1f548c-6e14-44b6-89da-9e1fed3c5a8c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:54.885057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 713/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:11:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:54.891356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75323ba-4b46-4910-9869-5544c2155d45 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:54.891806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75323ba-4b46-4910-9869-5544c2155d45 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:54.891914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75323ba-4b46-4910-9869-5544c2155d45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:54.966451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b75323ba-4b46-4910-9869-5544c2155d45 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:54.967681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b75323ba-4b46-4910-9869-5544c2155d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:54.969687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:55.069424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 726/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:11:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:11:55.220655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1f548c-6e14-44b6-89da-9e1fed3c5a8c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:55.222185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea1f548c-6e14-44b6-89da-9e1fed3c5a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:55.363559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1f548c-6e14-44b6-89da-9e1fed3c5a8c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:55.391875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:55.479792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4643ca-5373-41f8-9c5f-0ea6f3b2cb2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:55.480319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4643ca-5373-41f8-9c5f-0ea6f3b2cb2e service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:55.480602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4643ca-5373-41f8-9c5f-0ea6f3b2cb2e service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:55.480866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4643ca-5373-41f8-9c5f-0ea6f3b2cb2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:55.482479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5b4643ca-5373-41f8-9c5f-0ea6f3b2cb2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:55.571960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4643ca-5373-41f8-9c5f-0ea6f3b2cb2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:55.573661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4643ca-5373-41f8-9c5f-0ea6f3b2cb2e service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:55.592944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 714/1440] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:11:57.680845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d5c8d-8d10-4067-bfc0-316af4a8c63c None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:57.681455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d5c8d-8d10-4067-bfc0-316af4a8c63c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:57.681455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d5c8d-8d10-4067-bfc0-316af4a8c63c None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:57.682764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 727/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:11:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:11:57.688120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72730d7-51aa-4dd7-8db3-f33f6e7564e5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:57.688556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72730d7-51aa-4dd7-8db3-f33f6e7564e5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:57.688664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72730d7-51aa-4dd7-8db3-f33f6e7564e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:57.731856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:57.757550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c72730d7-51aa-4dd7-8db3-f33f6e7564e5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:11:57.758745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c72730d7-51aa-4dd7-8db3-f33f6e7564e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:57.856610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 715/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:11:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:11:58.145385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9d5c8d-8d10-4067-bfc0-316af4a8c63c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:11:58.146887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ef9d5c8d-8d10-4067-bfc0-316af4a8c63c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:58.247788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9d5c8d-8d10-4067-bfc0-316af4a8c63c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:11:58.280449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:11:58.323396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aab60f-bce4-44b5-a10a-ff00d54348c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:11:58.323994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aab60f-bce4-44b5-a10a-ff00d54348c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:11:58.324270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aab60f-bce4-44b5-a10a-ff00d54348c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:11:58.324533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aab60f-bce4-44b5-a10a-ff00d54348c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:11:58.326126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c8aab60f-bce4-44b5-a10a-ff00d54348c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:11:58.432147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aab60f-bce4-44b5-a10a-ff00d54348c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:11:58.433609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aab60f-bce4-44b5-a10a-ff00d54348c9 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:11:58.452170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 728/1443] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:11:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:00.744380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5956a2-5d51-4d3d-993e-c11636048e2f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:00.744984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5956a2-5d51-4d3d-993e-c11636048e2f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:00.744984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5956a2-5d51-4d3d-993e-c11636048e2f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:00.746321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 716/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:00.751556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fdd691-a728-4481-8549-e182d738cd53 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:00.751952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fdd691-a728-4481-8549-e182d738cd53 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:00.752090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fdd691-a728-4481-8549-e182d738cd53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:00.816407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-80fdd691-a728-4481-8549-e182d738cd53 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:00.818087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-80fdd691-a728-4481-8549-e182d738cd53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:00.830379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:00.914037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 729/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:01.203352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5956a2-5d51-4d3d-993e-c11636048e2f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:01.204884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5c5956a2-5d51-4d3d-993e-c11636048e2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:01.314728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5956a2-5d51-4d3d-993e-c11636048e2f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:01.328720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:01.395761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532b9b29-1d58-4b04-8d1c-0d90e5b443e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:01.396371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b9b29-1d58-4b04-8d1c-0d90e5b443e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:01.396649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b9b29-1d58-4b04-8d1c-0d90e5b443e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:01.396919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532b9b29-1d58-4b04-8d1c-0d90e5b443e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:01.398618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-532b9b29-1d58-4b04-8d1c-0d90e5b443e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:01.508492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532b9b29-1d58-4b04-8d1c-0d90e5b443e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:01.509900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532b9b29-1d58-4b04-8d1c-0d90e5b443e5 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:01.527495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 717/1446] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:03.994351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab7419-a0ad-4e75-b752-ee0076af14a7 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:03.994945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab7419-a0ad-4e75-b752-ee0076af14a7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:03.994945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab7419-a0ad-4e75-b752-ee0076af14a7 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:03.996259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 730/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:04.001588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76ebe55-819d-4342-9166-6e86f84870bc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:04.002008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76ebe55-819d-4342-9166-6e86f84870bc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:04.002139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76ebe55-819d-4342-9166-6e86f84870bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:04.050652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:04.068986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a76ebe55-819d-4342-9166-6e86f84870bc None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:04.070221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a76ebe55-819d-4342-9166-6e86f84870bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:04.167579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 718/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:04.458700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ab7419-a0ad-4e75-b752-ee0076af14a7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:04.460231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f5ab7419-a0ad-4e75-b752-ee0076af14a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:04.567484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ab7419-a0ad-4e75-b752-ee0076af14a7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:04.606744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:04.641077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eb59f9-9755-4094-87f6-1533ece71071 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:04.641680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eb59f9-9755-4094-87f6-1533ece71071 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:04.641961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eb59f9-9755-4094-87f6-1533ece71071 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:04.642233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eb59f9-9755-4094-87f6-1533ece71071 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:04.643829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b8eb59f9-9755-4094-87f6-1533ece71071 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:04.751684 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eb59f9-9755-4094-87f6-1533ece71071 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:04.753071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8eb59f9-9755-4094-87f6-1533ece71071 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:04.771759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 731/1449] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:07.074045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061443d-9e36-487a-b7fb-bef96fa5a69f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:07.074592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061443d-9e36-487a-b7fb-bef96fa5a69f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:07.074707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061443d-9e36-487a-b7fb-bef96fa5a69f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:07.076286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 719/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:07.082036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd871db-22c6-474a-8e63-99dcde60daeb None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:07.083170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd871db-22c6-474a-8e63-99dcde60daeb None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:07.083506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd871db-22c6-474a-8e63-99dcde60daeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:07.150071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5cd871db-22c6-474a-8e63-99dcde60daeb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:07.151272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5cd871db-22c6-474a-8e63-99dcde60daeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:07.173838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:07.250347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 732/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:07.539754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2061443d-9e36-487a-b7fb-bef96fa5a69f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:07.541311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2061443d-9e36-487a-b7fb-bef96fa5a69f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:07.645522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2061443d-9e36-487a-b7fb-bef96fa5a69f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:07.669379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:07.721919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5221f69-69db-4952-8542-1d74e5ef188d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:07.722519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5221f69-69db-4952-8542-1d74e5ef188d service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:07.722816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5221f69-69db-4952-8542-1d74e5ef188d service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:07.723732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5221f69-69db-4952-8542-1d74e5ef188d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:07.725574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d5221f69-69db-4952-8542-1d74e5ef188d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:07.833889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5221f69-69db-4952-8542-1d74e5ef188d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:07.835324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5221f69-69db-4952-8542-1d74e5ef188d service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:07.853273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 720/1452] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:10.165714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e13585-41a4-4e26-978f-6fd31fe15b75 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:10.166287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e13585-41a4-4e26-978f-6fd31fe15b75 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:10.166287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e13585-41a4-4e26-978f-6fd31fe15b75 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:10.167585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 733/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:10.172970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc28c56-0d8a-46b4-89ba-940e431d9ab3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:10.173361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc28c56-0d8a-46b4-89ba-940e431d9ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:10.173518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc28c56-0d8a-46b4-89ba-940e431d9ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:10.237171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:10.240567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1fc28c56-0d8a-46b4-89ba-940e431d9ab3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:10.242212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1fc28c56-0d8a-46b4-89ba-940e431d9ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:10.337537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 721/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:10.628716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e13585-41a4-4e26-978f-6fd31fe15b75 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:10.630315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a0e13585-41a4-4e26-978f-6fd31fe15b75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:10.731573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e13585-41a4-4e26-978f-6fd31fe15b75 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:10.779831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:10.805999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fe5a81-5334-4bda-9f01-fd015078910c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:10.806605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe5a81-5334-4bda-9f01-fd015078910c service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:10.806882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe5a81-5334-4bda-9f01-fd015078910c service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:10.807151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe5a81-5334-4bda-9f01-fd015078910c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:10.808723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-80fe5a81-5334-4bda-9f01-fd015078910c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:10.919188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fe5a81-5334-4bda-9f01-fd015078910c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:10.920655 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fe5a81-5334-4bda-9f01-fd015078910c service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:10.939208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 734/1455] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:13.164128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dfdfba-39a8-4c2e-bd5b-fe9de9d0ef4c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:13.164692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dfdfba-39a8-4c2e-bd5b-fe9de9d0ef4c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:13.164692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dfdfba-39a8-4c2e-bd5b-fe9de9d0ef4c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:13.165960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 722/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:12:13.171405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad03fc-f514-4622-9279-6490fb8d505c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:13.171828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad03fc-f514-4622-9279-6490fb8d505c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:13.171954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad03fc-f514-4622-9279-6490fb8d505c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:13.237902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e7ad03fc-f514-4622-9279-6490fb8d505c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:13.239878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e7ad03fc-f514-4622-9279-6490fb8d505c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:13.268846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:13.337125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 735/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:13.624109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dfdfba-39a8-4c2e-bd5b-fe9de9d0ef4c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:13.625722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-57dfdfba-39a8-4c2e-bd5b-fe9de9d0ef4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:13.726938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dfdfba-39a8-4c2e-bd5b-fe9de9d0ef4c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:13.755273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:13.799971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056041ab-1eab-4e7b-879f-a54a538a69ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:13.800504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056041ab-1eab-4e7b-879f-a54a538a69ac service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:13.800782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056041ab-1eab-4e7b-879f-a54a538a69ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:13.801712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056041ab-1eab-4e7b-879f-a54a538a69ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:13.803299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-056041ab-1eab-4e7b-879f-a54a538a69ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:13.914944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056041ab-1eab-4e7b-879f-a54a538a69ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:13.916372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056041ab-1eab-4e7b-879f-a54a538a69ac service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:13.934194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 723/1458] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:16.171822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e004452-c879-4594-ac56-bcaaaa0d2418 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:16.172406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e004452-c879-4594-ac56-bcaaaa0d2418 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:16.172406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e004452-c879-4594-ac56-bcaaaa0d2418 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:16.173779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 736/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:16.180185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e3cff-3c51-4f69-9ae5-7af84cefd5c0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:16.180707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e3cff-3c51-4f69-9ae5-7af84cefd5c0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:16.180981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e3cff-3c51-4f69-9ae5-7af84cefd5c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:16.247484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b63e3cff-3c51-4f69-9ae5-7af84cefd5c0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:16.248707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b63e3cff-3c51-4f69-9ae5-7af84cefd5c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:16.257486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:16.346436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 724/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:16.638478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e004452-c879-4594-ac56-bcaaaa0d2418 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:16.639997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5e004452-c879-4594-ac56-bcaaaa0d2418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:16.740944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e004452-c879-4594-ac56-bcaaaa0d2418 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:16.795802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:16.813608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c34f77-8be8-41ee-9bda-3cf2a045d4a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:16.814230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c34f77-8be8-41ee-9bda-3cf2a045d4a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:16.814504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c34f77-8be8-41ee-9bda-3cf2a045d4a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:16.814769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c34f77-8be8-41ee-9bda-3cf2a045d4a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:16.816333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-49c34f77-8be8-41ee-9bda-3cf2a045d4a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:16.931696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c34f77-8be8-41ee-9bda-3cf2a045d4a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:16.933872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c34f77-8be8-41ee-9bda-3cf2a045d4a0 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:16.954007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 737/1461] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:19.147068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148faf32-47db-4eca-b02f-0754dfd50f5d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:19.147686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148faf32-47db-4eca-b02f-0754dfd50f5d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:19.147686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148faf32-47db-4eca-b02f-0754dfd50f5d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:19.148996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 725/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:19.154709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064fb492-8364-4824-88e6-7aa5f29a4904 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:19.155134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064fb492-8364-4824-88e6-7aa5f29a4904 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:19.155276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064fb492-8364-4824-88e6-7aa5f29a4904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:19.221010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-064fb492-8364-4824-88e6-7aa5f29a4904 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:19.222216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-064fb492-8364-4824-88e6-7aa5f29a4904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:19.260631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:19.318860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 738/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:19.606813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148faf32-47db-4eca-b02f-0754dfd50f5d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:19.608356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-148faf32-47db-4eca-b02f-0754dfd50f5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:19.710274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148faf32-47db-4eca-b02f-0754dfd50f5d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:19.746675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:19.783810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a8bb14-d125-4978-9edf-8598ca13e1b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:19.784379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a8bb14-d125-4978-9edf-8598ca13e1b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:19.784659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a8bb14-d125-4978-9edf-8598ca13e1b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:19.784952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a8bb14-d125-4978-9edf-8598ca13e1b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:19.786907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-96a8bb14-d125-4978-9edf-8598ca13e1b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:19.893411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a8bb14-d125-4978-9edf-8598ca13e1b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:19.895598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a8bb14-d125-4978-9edf-8598ca13e1b5 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:19.916952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 726/1464] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:12:22.103199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18cbb7a-7c4b-4dad-83a2-fcbda090b49b None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:22.103771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18cbb7a-7c4b-4dad-83a2-fcbda090b49b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:22.103771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18cbb7a-7c4b-4dad-83a2-fcbda090b49b None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:22.105119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 739/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:22.110537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdffc85a-f173-4af0-96ac-684f30a00d6e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:22.111004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdffc85a-f173-4af0-96ac-684f30a00d6e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:22.111109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdffc85a-f173-4af0-96ac-684f30a00d6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:22.178992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-bdffc85a-f173-4af0-96ac-684f30a00d6e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:22.180200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bdffc85a-f173-4af0-96ac-684f30a00d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:22.198361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:22.284121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 727/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:22.583754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18cbb7a-7c4b-4dad-83a2-fcbda090b49b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:22.585324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d18cbb7a-7c4b-4dad-83a2-fcbda090b49b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:22.687953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18cbb7a-7c4b-4dad-83a2-fcbda090b49b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:22.753362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:22.759920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4477a619-25ef-48b6-96ff-8ef5d5429fb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:22.760516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4477a619-25ef-48b6-96ff-8ef5d5429fb2 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:22.760803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4477a619-25ef-48b6-96ff-8ef5d5429fb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:22.761088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4477a619-25ef-48b6-96ff-8ef5d5429fb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:22.762646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4477a619-25ef-48b6-96ff-8ef5d5429fb2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:22.868142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4477a619-25ef-48b6-96ff-8ef5d5429fb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:22.869523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4477a619-25ef-48b6-96ff-8ef5d5429fb2 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:22.886837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 740/1467] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:25.058962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9588fdc3-4ab5-4a1a-b82e-3293535f6f14 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:25.059520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9588fdc3-4ab5-4a1a-b82e-3293535f6f14 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:25.059520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9588fdc3-4ab5-4a1a-b82e-3293535f6f14 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:25.060833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 728/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:12:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:25.066136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb5076-961a-4337-bc33-79baaabfb6d4 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:25.067108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb5076-961a-4337-bc33-79baaabfb6d4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:25.067288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb5076-961a-4337-bc33-79baaabfb6d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:25.132594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-6fdb5076-961a-4337-bc33-79baaabfb6d4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:25.133831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6fdb5076-961a-4337-bc33-79baaabfb6d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:25.182926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:25.231770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 741/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:12:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:12:25.517901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9588fdc3-4ab5-4a1a-b82e-3293535f6f14 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:25.519520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9588fdc3-4ab5-4a1a-b82e-3293535f6f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:25.624917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9588fdc3-4ab5-4a1a-b82e-3293535f6f14 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:25.672413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:25.698019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0ab589-0a63-4289-840c-34dca802d587 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:25.698619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ab589-0a63-4289-840c-34dca802d587 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:25.698892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ab589-0a63-4289-840c-34dca802d587 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:25.699158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0ab589-0a63-4289-840c-34dca802d587 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:25.700718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0a0ab589-0a63-4289-840c-34dca802d587 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:25.804770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0ab589-0a63-4289-840c-34dca802d587 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:25.806226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0ab589-0a63-4289-840c-34dca802d587 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:25.823840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 729/1470] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:35.230298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=22 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:35.821731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=66 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:45.851593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb61b955-d947-445f-8070-cea2a25e3123 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:45.852160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb61b955-d947-445f-8070-cea2a25e3123 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:45.852160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb61b955-d947-445f-8070-cea2a25e3123 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:45.853654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 742/1471] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:12:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:45.858902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e15867-70ce-4862-9e67-4d24c19dc736 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:45.859316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e15867-70ce-4862-9e67-4d24c19dc736 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:45.859435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e15867-70ce-4862-9e67-4d24c19dc736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:45.926907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-48e15867-70ce-4862-9e67-4d24c19dc736 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:45.928106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-48e15867-70ce-4862-9e67-4d24c19dc736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:46.025358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 730/1472] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:12:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:46.326251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf3cbdb-a9f5-4268-a4a2-9166bdd5d8b1 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:46.326534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf3cbdb-a9f5-4268-a4a2-9166bdd5d8b1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:46.326807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf3cbdb-a9f5-4268-a4a2-9166bdd5d8b1 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:46.328197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 743/1473] 10.222.0.26 () {54 vars in 962 bytes} [Sat Oct 1 04:12:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:46.333365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9159de64-3f85-46eb-89a8-f3b9c6847988 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:46.333655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9159de64-3f85-46eb-89a8-f3b9c6847988 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:46.333930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9159de64-3f85-46eb-89a8-f3b9c6847988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:46.408717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9159de64-3f85-46eb-89a8-f3b9c6847988 None None] MFA Rules not processed for user `170fe0394f3d4472a1380481cfe2cc28`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:46.410042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9159de64-3f85-46eb-89a8-f3b9c6847988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:46.430254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:46.484296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 731/1474] 10.222.0.26 () {58 vars in 1059 bytes} [Sat Oct 1 04:12:46 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:46.493116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf3cbdb-a9f5-4268-a4a2-9166bdd5d8b1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:46.494600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-bdf3cbdb-a9f5-4268-a4a2-9166bdd5d8b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:46.598498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf3cbdb-a9f5-4268-a4a2-9166bdd5d8b1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:46.670530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd42e34-2b57-438a-bff2-d9426e242f8f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:46.671130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd42e34-2b57-438a-bff2-d9426e242f8f service glance] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:46.671409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd42e34-2b57-438a-bff2-d9426e242f8f service glance] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:46.671700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd42e34-2b57-438a-bff2-d9426e242f8f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:46.673303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-dbd42e34-2b57-438a-bff2-d9426e242f8f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:46.726717 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:46.782358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd42e34-2b57-438a-bff2-d9426e242f8f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:46.783708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd42e34-2b57-438a-bff2-d9426e242f8f service glance] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:46.806722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 744/1475] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:49.182131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b782b-c339-4c23-990c-4b3f8bd49752 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:49.182729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b782b-c339-4c23-990c-4b3f8bd49752 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:49.182729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b782b-c339-4c23-990c-4b3f8bd49752 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:49.184025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 732/1476] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:12:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:49.189354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da1f2fe-69de-406a-ac87-3ebc506b7486 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:49.189783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da1f2fe-69de-406a-ac87-3ebc506b7486 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:49.189885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da1f2fe-69de-406a-ac87-3ebc506b7486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:49.254876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-3da1f2fe-69de-406a-ac87-3ebc506b7486 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:49.255985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3da1f2fe-69de-406a-ac87-3ebc506b7486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:49.348868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 745/1477] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:12:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:12:49.622427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4b782b-c339-4c23-990c-4b3f8bd49752 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:49.623949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0a4b782b-c339-4c23-990c-4b3f8bd49752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:49.747421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4b782b-c339-4c23-990c-4b3f8bd49752 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:49.797230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:49.840912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73119e01-3f9b-4f53-9596-5f5e6ef92e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:49.841543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73119e01-3f9b-4f53-9596-5f5e6ef92e36 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:49.841816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73119e01-3f9b-4f53-9596-5f5e6ef92e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:49.842081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73119e01-3f9b-4f53-9596-5f5e6ef92e36 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:49.842709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73119e01-3f9b-4f53-9596-5f5e6ef92e36 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:49.844419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-73119e01-3f9b-4f53-9596-5f5e6ef92e36 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:12:49.844419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:12:49.844419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:12:49.845600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73119e01-3f9b-4f53-9596-5f5e6ef92e36 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:49.864213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 733/1478] 10.222.0.26 () {56 vars in 1108 bytes} [Sat Oct 1 04:12:49 2022] GET /identity/v3/projects => generated 2553 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:12:52.021952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdbb241-13fe-49f6-b50f-459e3b03a79e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:52.022582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdbb241-13fe-49f6-b50f-459e3b03a79e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:52.022582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdbb241-13fe-49f6-b50f-459e3b03a79e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:52.024175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 746/1479] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:12:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:52.029924 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa953-c0ed-4b9a-ba35-1d495469a7b3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:52.030370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa953-c0ed-4b9a-ba35-1d495469a7b3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:52.030476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa953-c0ed-4b9a-ba35-1d495469a7b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:52.095472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-21faa953-c0ed-4b9a-ba35-1d495469a7b3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:52.096673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-21faa953-c0ed-4b9a-ba35-1d495469a7b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:52.193500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 734/1480] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:12:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:12:52.485450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdbb241-13fe-49f6-b50f-459e3b03a79e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:52.487015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0cdbb241-13fe-49f6-b50f-459e3b03a79e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:52.497354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:52.629958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdbb241-13fe-49f6-b50f-459e3b03a79e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:52.709451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:52.743417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5dbde0-87a9-47c0-8dff-7086e0ddc985 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:52.743935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5dbde0-87a9-47c0-8dff-7086e0ddc985 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:52.744213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5dbde0-87a9-47c0-8dff-7086e0ddc985 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:52.744484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5dbde0-87a9-47c0-8dff-7086e0ddc985 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:52.746110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9c5dbde0-87a9-47c0-8dff-7086e0ddc985 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:52.836034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5dbde0-87a9-47c0-8dff-7086e0ddc985 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:52.837652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5dbde0-87a9-47c0-8dff-7086e0ddc985 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:52.855122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 747/1481] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:55.187142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c62cc8-7e4c-4faa-abae-b29aba81457d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:55.187722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c62cc8-7e4c-4faa-abae-b29aba81457d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:55.187722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c62cc8-7e4c-4faa-abae-b29aba81457d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:55.189041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 735/1482] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:12:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:55.195093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1640f-2acb-4888-ae97-f792d2224705 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:55.195499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1640f-2acb-4888-ae97-f792d2224705 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:55.195669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1640f-2acb-4888-ae97-f792d2224705 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:55.261159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-16d1640f-2acb-4888-ae97-f792d2224705 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:55.262367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-16d1640f-2acb-4888-ae97-f792d2224705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:55.309501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:55.358930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 748/1483] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:55.660494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c62cc8-7e4c-4faa-abae-b29aba81457d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:55.662144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a7c62cc8-7e4c-4faa-abae-b29aba81457d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:55.685582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:55.764556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c62cc8-7e4c-4faa-abae-b29aba81457d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:55.836353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da758319-8dcf-425a-907d-ac58306f3548 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:55.836964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da758319-8dcf-425a-907d-ac58306f3548 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:55.837247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da758319-8dcf-425a-907d-ac58306f3548 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:55.837515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da758319-8dcf-425a-907d-ac58306f3548 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:55.839072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-da758319-8dcf-425a-907d-ac58306f3548 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:55.911108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:55.945981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da758319-8dcf-425a-907d-ac58306f3548 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:55.947413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da758319-8dcf-425a-907d-ac58306f3548 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:55.965112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 736/1484] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:12:58.167162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1ec98f-9b80-4c84-8dae-d546e0eaeded None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:58.167691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1ec98f-9b80-4c84-8dae-d546e0eaeded None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:58.167691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1ec98f-9b80-4c84-8dae-d546e0eaeded None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:58.168998 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 749/1485] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:12:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:12:58.174401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc78d1-a4c1-4665-9ca1-05d0befefa99 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:58.175696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc78d1-a4c1-4665-9ca1-05d0befefa99 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:58.176010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dc78d1-a4c1-4665-9ca1-05d0befefa99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:58.243034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-96dc78d1-a4c1-4665-9ca1-05d0befefa99 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:12:58.244243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-96dc78d1-a4c1-4665-9ca1-05d0befefa99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:58.340204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 737/1486] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:12:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:12:58.641893 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1ec98f-9b80-4c84-8dae-d546e0eaeded None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:12:58.643427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0a1ec98f-9b80-4c84-8dae-d546e0eaeded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:58.749807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1ec98f-9b80-4c84-8dae-d546e0eaeded None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:12:58.822238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:12:58.823678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb55eb15-b07e-40c4-90a5-e08d8632f367 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:12:58.824406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb55eb15-b07e-40c4-90a5-e08d8632f367 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:12:58.824698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb55eb15-b07e-40c4-90a5-e08d8632f367 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:12:58.825189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb55eb15-b07e-40c4-90a5-e08d8632f367 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:12:58.827138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cb55eb15-b07e-40c4-90a5-e08d8632f367 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:12:58.933305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb55eb15-b07e-40c4-90a5-e08d8632f367 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:12:58.934721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb55eb15-b07e-40c4-90a5-e08d8632f367 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:12:58.952576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 750/1487] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:12:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:13:01.207561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4088581b-4e11-4e12-b627-9267dd8bb467 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:13:01.208106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4088581b-4e11-4e12-b627-9267dd8bb467 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:13:01.208106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4088581b-4e11-4e12-b627-9267dd8bb467 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:13:01.209492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 738/1488] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:13:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:13:01.215038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8754afd0-c73f-4067-b238-df5c69416c3c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:13:01.215457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8754afd0-c73f-4067-b238-df5c69416c3c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:13:01.215590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8754afd0-c73f-4067-b238-df5c69416c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:13:01.285973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8754afd0-c73f-4067-b238-df5c69416c3c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:13:01.287170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8754afd0-c73f-4067-b238-df5c69416c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:13:01.339346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:13:01.383114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 751/1489] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:13:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:13:01.674185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4088581b-4e11-4e12-b627-9267dd8bb467 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:13:01.676242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4088581b-4e11-4e12-b627-9267dd8bb467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:13:01.713354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:13:01.780814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4088581b-4e11-4e12-b627-9267dd8bb467 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:13:01.859056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536d334d-d7b3-4fbc-ad30-c239436b2191 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:13:01.859661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d334d-d7b3-4fbc-ad30-c239436b2191 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:13:01.859938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d334d-d7b3-4fbc-ad30-c239436b2191 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:13:01.860208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d334d-d7b3-4fbc-ad30-c239436b2191 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:13:01.861831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-536d334d-d7b3-4fbc-ad30-c239436b2191 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:13:01.940781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:13:01.970317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536d334d-d7b3-4fbc-ad30-c239436b2191 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:13:01.972773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536d334d-d7b3-4fbc-ad30-c239436b2191 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:13:01.990694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 739/1490] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:13:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:13:04.352830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f983a-75ef-44de-8a1f-80354d18c36e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:13:04.353454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f983a-75ef-44de-8a1f-80354d18c36e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:13:04.353454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0f983a-75ef-44de-8a1f-80354d18c36e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:13:04.354731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 752/1491] 10.222.0.26 () {54 vars in 927 bytes} [Sat Oct 1 04:13:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:13:04.360065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0278a1-cc2f-43d4-a195-6e961dedaf6a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:13:04.360499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0278a1-cc2f-43d4-a195-6e961dedaf6a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:13:04.360610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0278a1-cc2f-43d4-a195-6e961dedaf6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:13:04.428278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1c0278a1-cc2f-43d4-a195-6e961dedaf6a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:13:04.429516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1c0278a1-cc2f-43d4-a195-6e961dedaf6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:13:04.525158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 740/1492] 10.222.0.26 () {58 vars in 1024 bytes} [Sat Oct 1 04:13:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:13:04.820151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0f983a-75ef-44de-8a1f-80354d18c36e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:13:04.821718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2f0f983a-75ef-44de-8a1f-80354d18c36e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:13:04.922345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0f983a-75ef-44de-8a1f-80354d18c36e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:13:04.994818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db51f065-ec34-4aa9-8c4a-7b5f54f469eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:13:04.995441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db51f065-ec34-4aa9-8c4a-7b5f54f469eb service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:13:04.995714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db51f065-ec34-4aa9-8c4a-7b5f54f469eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:13:04.995979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db51f065-ec34-4aa9-8c4a-7b5f54f469eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:13:04.997567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-db51f065-ec34-4aa9-8c4a-7b5f54f469eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:13:05.006158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:13:05.106792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db51f065-ec34-4aa9-8c4a-7b5f54f469eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:13:05.108131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db51f065-ec34-4aa9-8c4a-7b5f54f469eb service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:13:05.125977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 753/1493] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:13:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:13:14.523223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=82 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:13:15.123687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=50 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:18.311679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:18.312279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:18.312279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:18.386037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:14:18.387219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:18.481665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 741/1494] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:14:18.490085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c637a09-ef4c-47e5-ba1a-32298aca04b5 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:18.490590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c637a09-ef4c-47e5-ba1a-32298aca04b5 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:18.490719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c637a09-ef4c-47e5-ba1a-32298aca04b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:18.570615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-3c637a09-ef4c-47e5-ba1a-32298aca04b5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:14:18.571719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3c637a09-ef4c-47e5-ba1a-32298aca04b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:18.665623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 754/1495] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:14:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:14:18.670209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:18.671439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:18.682872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624408e1-f2a4-471b-92cc-74fff1b3d455 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:18.743461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:18.777015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:18.777735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:18.778014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:18.778284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:18.779190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:14:18.779491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:18.780486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:18.780486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:14:18.780486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:14:18.781409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:18.793246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 742/1496] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:14:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:14:18.801866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c637a09-ef4c-47e5-ba1a-32298aca04b5 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:18.810959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c637a09-ef4c-47e5-ba1a-32298aca04b5 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:18.871441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:18.905543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:18.906145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:18.906422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:18.906692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:18.907560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:18.909342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING py.warnings [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:18.909342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: warnings.warn(msg) Oct 01 04:14:18.909342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: Oct 01 04:14:18.910539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:18.966113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 755/1497] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:14:18 2022] POST /identity/v3/projects => generated 379 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:14:18.970338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:18.971833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:19.004616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e723ae71-a4d3-4f19-ac31-c42c78ab98de admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:19.097275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:19.097881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:19.098161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:19.098433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:19.099074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:19.100069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:19.100069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:14:19.100069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:14:19.101306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:19.145010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 743/1498] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:14:18 2022] POST /identity/v3/users => generated 290 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:14:19.149077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:19.150572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:19.212418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c366d1-efd9-415f-8974-bba210f8f5b9 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:19.295102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:19.307073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:19.307673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:19.307954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:19.308224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:19.308866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:19.309885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING py.warnings [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:19.309885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: warnings.warn(msg) Oct 01 04:14:19.309885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: Oct 01 04:14:19.310776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:19.323924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 756/1499] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:14:19.328118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:19.329662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:19.340706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a77d1cd-9173-4dcd-8773-413c648da6ee admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:19.366527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:19.433982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63319bda-7cf2-4aa4-8a04-b81c81768bc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:19.434581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63319bda-7cf2-4aa4-8a04-b81c81768bc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:19.434855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63319bda-7cf2-4aa4-8a04-b81c81768bc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:19.435124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63319bda-7cf2-4aa4-8a04-b81c81768bc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:19.435755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63319bda-7cf2-4aa4-8a04-b81c81768bc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:19.436742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-63319bda-7cf2-4aa4-8a04-b81c81768bc5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:19.436742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:14:19.436742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:14:19.437656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63319bda-7cf2-4aa4-8a04-b81c81768bc5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:19.451186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 744/1500] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:14:19.455563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:19.457120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:19.490158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e1c10d-7d7d-48ad-a9e5-504d53a69d4e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:19.585577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8029c6e-bd07-44c8-ba3d-376956ae4238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:19.586224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8029c6e-bd07-44c8-ba3d-376956ae4238 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:19.586502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8029c6e-bd07-44c8-ba3d-376956ae4238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:19.586771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8029c6e-bd07-44c8-ba3d-376956ae4238 admin admin] PATH_INFO: `/v3/projects/61be4973daa844309b37815ce59a1e0d/users/8809f08eea474a8e8ab85af28dc2e502/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:19.608248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8029c6e-bd07-44c8-ba3d-376956ae4238 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61be4973daa844309b37815ce59a1e0d, user_id=8809f08eea474a8e8ab85af28dc2e502, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:19.609287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING py.warnings [None req-b8029c6e-bd07-44c8-ba3d-376956ae4238 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:19.609287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: warnings.warn(msg) Oct 01 04:14:19.609287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: Oct 01 04:14:19.611465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8029c6e-bd07-44c8-ba3d-376956ae4238 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:19.635039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 757/1501] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:14:19 2022] PUT /identity/v3/projects/61be4973daa844309b37815ce59a1e0d/users/8809f08eea474a8e8ab85af28dc2e502/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:14:19.648049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902261e-41f1-4077-aa00-2c070d00bdbc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:19.648354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902261e-41f1-4077-aa00-2c070d00bdbc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:19.648613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902261e-41f1-4077-aa00-2c070d00bdbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:19.706899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-2902261e-41f1-4077-aa00-2c070d00bdbc None None] MFA Rules not processed for user `8809f08eea474a8e8ab85af28dc2e502`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:14:19.708029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2902261e-41f1-4077-aa00-2c070d00bdbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:19.786732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 745/1502] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:14:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:14:19.800160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6422692-ce18-440d-bc13-1ffa2373a172 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:19.800452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6422692-ce18-440d-bc13-1ffa2373a172 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:19.800729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6422692-ce18-440d-bc13-1ffa2373a172 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:19.802107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 758/1503] 10.222.0.26 () {54 vars in 963 bytes} [Sat Oct 1 04:14:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:14:19.806786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e3840-46c7-4282-a022-6fe3971c3cad None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:19.807076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e3840-46c7-4282-a022-6fe3971c3cad None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:19.807353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e3840-46c7-4282-a022-6fe3971c3cad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:19.839149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:19.883148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fe8e3840-46c7-4282-a022-6fe3971c3cad None None] MFA Rules not processed for user `52b5d64c999a4f16afec678df3025529`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:14:19.884347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fe8e3840-46c7-4282-a022-6fe3971c3cad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:19.958208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 746/1504] 10.222.0.26 () {58 vars in 1060 bytes} [Sat Oct 1 04:14:19 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:14:19.966225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6422692-ce18-440d-bc13-1ffa2373a172 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:19.967734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f6422692-ce18-440d-bc13-1ffa2373a172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:20.019129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:20.084722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6422692-ce18-440d-bc13-1ffa2373a172 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:20.158518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:20.159123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:20.159403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:20.159678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:20.161299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:20.260454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:20.261826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:20.265336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:20.280859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 759/1505] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:14:19 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:14:20.326455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2744e11-676e-47e4-9dee-592252164c3f None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:20.326768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2744e11-676e-47e4-9dee-592252164c3f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:20.327046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2744e11-676e-47e4-9dee-592252164c3f None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:20.328280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 747/1506] 10.222.0.26 () {52 vars in 830 bytes} [Sat Oct 1 04:14:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:14:20.331994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:20.333210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:20.433185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb20fd5-e006-45c2-9afc-499afcec23cd service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:20.529755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:20.530390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:20.530685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:20.530938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] PATH_INFO: `/v3/users/8809f08eea474a8e8ab85af28dc2e502` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:20.543756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8809f08eea474a8e8ab85af28dc2e502)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:20.544764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING py.warnings [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:20.544764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: warnings.warn(msg) Oct 01 04:14:20.544764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: Oct 01 04:14:20.545998 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:20.555610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:20.611506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8809f08eea474a8e8ab85af28dc2e502'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:20.611940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8809f08eea474a8e8ab85af28dc2e502'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:20.625389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8809f08eea474a8e8ab85af28dc2e502'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:20.626647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8809f08eea474a8e8ab85af28dc2e502'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:20.635736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8809f08eea474a8e8ab85af28dc2e502'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:20.663704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 760/1507] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:14:20 2022] DELETE /identity/v3/users/8809f08eea474a8e8ab85af28dc2e502 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:14:20.668073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:20.668356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:20.668630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:20.727548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:14:20.728739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:20.803215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:20.816411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 748/1508] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:14:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:14:20.825991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:20.827183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:20.925980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81240f40-2334-42b3-97d2-c61176d27b4e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:20.979805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:20.999725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a306244-ce8b-4684-af4a-013b338dd875 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:21.000338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a306244-ce8b-4684-af4a-013b338dd875 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:21.001250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a306244-ce8b-4684-af4a-013b338dd875 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:21.001556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a306244-ce8b-4684-af4a-013b338dd875 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:21.003142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2a306244-ce8b-4684-af4a-013b338dd875 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:21.111486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a306244-ce8b-4684-af4a-013b338dd875 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:21.112833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a306244-ce8b-4684-af4a-013b338dd875 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:21.130319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 761/1509] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:14:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:14:21.490288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:14:21.492147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:14:21.507592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae89421-3b85-49b2-9375-e33a92527594 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:14:21.613577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:14:21.614262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:14:21.614490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:14:21.614765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] PATH_INFO: `/v3/projects/61be4973daa844309b37815ce59a1e0d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:14:21.625208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] RBAC: Authorizing `identity:delete_project(project_id=61be4973daa844309b37815ce59a1e0d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:14:21.626379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:14:21.626379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:14:21.626379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:14:21.627583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:14:21.684365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61be4973daa844309b37815ce59a1e0d', 'request_id': 'req-9ffc28c5-2b07-4315-808a-70ae4e0d7588'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:21.693728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61be4973daa844309b37815ce59a1e0d', 'request_id': 'req-9ffc28c5-2b07-4315-808a-70ae4e0d7588'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:21.707688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61be4973daa844309b37815ce59a1e0d', 'request_id': 'req-9ffc28c5-2b07-4315-808a-70ae4e0d7588'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:21.729963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] The token cache is being invalidate because project 61be4973daa844309b37815ce59a1e0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:14:21.730250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:21.730671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9ffc28c5-2b07-4315-808a-70ae4e0d7588 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:14:21.732068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 749/1510] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:14:21 2022] DELETE /identity/v3/projects/61be4973daa844309b37815ce59a1e0d => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:14:31.128070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=62 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:14:31.728076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:15:46.758043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdd3b0e-edb7-4c02-958a-445638a892b8 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:46.758642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdd3b0e-edb7-4c02-958a-445638a892b8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:46.758642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdd3b0e-edb7-4c02-958a-445638a892b8 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:46.759979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 762/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:15:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:15:46.764746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37fdfe1-9d50-4fc3-8ae5-b26de2324641 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:46.765188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37fdfe1-9d50-4fc3-8ae5-b26de2324641 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:46.765434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37fdfe1-9d50-4fc3-8ae5-b26de2324641 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:46.845191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f37fdfe1-9d50-4fc3-8ae5-b26de2324641 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:15:46.846309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f37fdfe1-9d50-4fc3-8ae5-b26de2324641 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:15:46.950394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 750/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:15:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:15:47.105864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdd3b0e-edb7-4c02-958a-445638a892b8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:15:47.107924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7cdd3b0e-edb7-4c02-958a-445638a892b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:15:47.229487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdd3b0e-edb7-4c02-958a-445638a892b8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:15:47.236079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f21d2a-cae6-4064-babe-7407b1c6b5d8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:15:47.236819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f21d2a-cae6-4064-babe-7407b1c6b5d8 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:47.237150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f21d2a-cae6-4064-babe-7407b1c6b5d8 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:47.237422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f21d2a-cae6-4064-babe-7407b1c6b5d8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:47.246120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f21d2a-cae6-4064-babe-7407b1c6b5d8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:15:47.247765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f21d2a-cae6-4064-babe-7407b1c6b5d8 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:15:47.257752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 763/1513] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:15:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:15:48.448145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735119d-0892-464a-a26f-c6a08dec167c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:48.448693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735119d-0892-464a-a26f-c6a08dec167c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:48.449002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735119d-0892-464a-a26f-c6a08dec167c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:48.450388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 751/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:15:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:15:48.457502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563530e8-be74-401f-982c-690fdba2240c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:48.458023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563530e8-be74-401f-982c-690fdba2240c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:48.458422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563530e8-be74-401f-982c-690fdba2240c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:48.486907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-563530e8-be74-401f-982c-690fdba2240c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:15:48.488472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-563530e8-be74-401f-982c-690fdba2240c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:15:48.498398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 764/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:15:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:15:48.675124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e735119d-0892-464a-a26f-c6a08dec167c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:15:48.689364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e735119d-0892-464a-a26f-c6a08dec167c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:15:48.694709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb975492-8724-416a-a6e5-17d11ea0aa2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:15:48.695311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb975492-8724-416a-a6e5-17d11ea0aa2b service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:48.695586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb975492-8724-416a-a6e5-17d11ea0aa2b service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:48.695859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb975492-8724-416a-a6e5-17d11ea0aa2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:48.703885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb975492-8724-416a-a6e5-17d11ea0aa2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:15:48.705427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb975492-8724-416a-a6e5-17d11ea0aa2b service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:15:48.714453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 752/1516] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:15:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:15:48.818769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd3b6c-5ea0-40c7-ba80-092196f98a48 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:48.819059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd3b6c-5ea0-40c7-ba80-092196f98a48 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:48.819338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd3b6c-5ea0-40c7-ba80-092196f98a48 None None] PATH_INFO: `/v3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:48.820645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 765/1517] 10.222.0.25 () {54 vars in 925 bytes} [Sat Oct 1 04:15:48 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:15:48.827342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf26f1-9b7a-417c-ab44-1b8f3ee2eafa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:48.827645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf26f1-9b7a-417c-ab44-1b8f3ee2eafa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:48.827961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf26f1-9b7a-417c-ab44-1b8f3ee2eafa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:48.892607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-24cf26f1-9b7a-417c-ab44-1b8f3ee2eafa None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:15:48.893864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-24cf26f1-9b7a-417c-ab44-1b8f3ee2eafa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:15:48.911086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:15:48.969616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 753/1518] 10.222.0.25 () {58 vars in 1015 bytes} [Sat Oct 1 04:15:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:15:52.558331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab42fbf-c0c9-49e0-88cf-4223a965d7a2 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:52.558842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab42fbf-c0c9-49e0-88cf-4223a965d7a2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:52.558968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab42fbf-c0c9-49e0-88cf-4223a965d7a2 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:52.560341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 766/1519] 10.222.0.26 () {54 vars in 961 bytes} [Sat Oct 1 04:15:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:15:52.566604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4aa84-4bec-40e3-a0c8-f4f93511ab2b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:52.567399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4aa84-4bec-40e3-a0c8-f4f93511ab2b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:52.567678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4aa84-4bec-40e3-a0c8-f4f93511ab2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:52.587493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c2c4aa84-4bec-40e3-a0c8-f4f93511ab2b None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:15:52.588674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c2c4aa84-4bec-40e3-a0c8-f4f93511ab2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:15:52.597626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 754/1520] 10.222.0.26 () {58 vars in 1058 bytes} [Sat Oct 1 04:15:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:15:52.605144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab42fbf-c0c9-49e0-88cf-4223a965d7a2 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:15:52.618999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab42fbf-c0c9-49e0-88cf-4223a965d7a2 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:15:52.623876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3489a14d-cae7-4138-8977-81ca7bdb336c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:15:52.624461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489a14d-cae7-4138-8977-81ca7bdb336c service placement] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:15:52.624869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489a14d-cae7-4138-8977-81ca7bdb336c service placement] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:15:52.625174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489a14d-cae7-4138-8977-81ca7bdb336c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:15:52.633584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3489a14d-cae7-4138-8977-81ca7bdb336c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:15:52.635150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3489a14d-cae7-4138-8977-81ca7bdb336c service placement] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:15:52.644633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 767/1521] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:15:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:16:00.200140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79484956-aef5-4994-b1c7-83e5f3b1c2fa None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:00.200752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79484956-aef5-4994-b1c7-83e5f3b1c2fa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:00.200752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79484956-aef5-4994-b1c7-83e5f3b1c2fa None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:00.202081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 755/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:16:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:16:00.208152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a15d6bd-707f-4b9d-ae55-20b550554776 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:00.208625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a15d6bd-707f-4b9d-ae55-20b550554776 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:00.208737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a15d6bd-707f-4b9d-ae55-20b550554776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:00.229079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2a15d6bd-707f-4b9d-ae55-20b550554776 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:16:00.230277 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2a15d6bd-707f-4b9d-ae55-20b550554776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:00.238760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 768/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:16:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:16:00.394840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79484956-aef5-4994-b1c7-83e5f3b1c2fa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:16:00.410394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79484956-aef5-4994-b1c7-83e5f3b1c2fa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:16:00.415862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de105833-509b-42ae-9659-2fea8039a1b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:16:00.416460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de105833-509b-42ae-9659-2fea8039a1b9 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:00.416769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de105833-509b-42ae-9659-2fea8039a1b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:00.417078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de105833-509b-42ae-9659-2fea8039a1b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:00.425015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de105833-509b-42ae-9659-2fea8039a1b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:16:00.426590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de105833-509b-42ae-9659-2fea8039a1b9 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:16:00.435709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 756/1524] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:16:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:16:01.530016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dad690-b5f4-4bee-8258-56b786665120 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:01.530491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dad690-b5f4-4bee-8258-56b786665120 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:01.530600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dad690-b5f4-4bee-8258-56b786665120 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:01.531918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 769/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sat Oct 1 04:16:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:16:01.538823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60226f39-78e3-4173-9b9b-24ded7d60684 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:01.539249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60226f39-78e3-4173-9b9b-24ded7d60684 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:01.539387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60226f39-78e3-4173-9b9b-24ded7d60684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:01.558978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-60226f39-78e3-4173-9b9b-24ded7d60684 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:16:01.560181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-60226f39-78e3-4173-9b9b-24ded7d60684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:01.568613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 757/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sat Oct 1 04:16:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:16:01.720007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dad690-b5f4-4bee-8258-56b786665120 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:16:01.734320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dad690-b5f4-4bee-8258-56b786665120 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:16:01.740886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5788c252-ec66-4cef-ad8c-73163da09c3d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:16:01.741522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788c252-ec66-4cef-ad8c-73163da09c3d service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:01.741805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788c252-ec66-4cef-ad8c-73163da09c3d service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:01.742081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788c252-ec66-4cef-ad8c-73163da09c3d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:01.751828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5788c252-ec66-4cef-ad8c-73163da09c3d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:16:01.753920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5788c252-ec66-4cef-ad8c-73163da09c3d service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:16:01.763375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 770/1527] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:16:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:16:05.963182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cb0cb3-1a85-4103-ae90-0b7ff2773b06 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:05.963731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cb0cb3-1a85-4103-ae90-0b7ff2773b06 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:05.963731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cb0cb3-1a85-4103-ae90-0b7ff2773b06 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:05.965164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 758/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:16:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:16:05.970298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab50ea85-8d53-4d29-a0d0-a1984b368bb3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:05.970750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab50ea85-8d53-4d29-a0d0-a1984b368bb3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:05.971403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab50ea85-8d53-4d29-a0d0-a1984b368bb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:05.990530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ab50ea85-8d53-4d29-a0d0-a1984b368bb3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:16:05.991726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ab50ea85-8d53-4d29-a0d0-a1984b368bb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:05.999456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 771/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:16:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:16:06.307687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cb0cb3-1a85-4103-ae90-0b7ff2773b06 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:16:06.309431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-81cb0cb3-1a85-4103-ae90-0b7ff2773b06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:06.390303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cb0cb3-1a85-4103-ae90-0b7ff2773b06 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:16:06.393993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266ff348-3119-4dda-bf81-ce96331e32eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:16:06.394587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266ff348-3119-4dda-bf81-ce96331e32eb service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:06.394870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266ff348-3119-4dda-bf81-ce96331e32eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:06.395138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266ff348-3119-4dda-bf81-ce96331e32eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:06.404070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266ff348-3119-4dda-bf81-ce96331e32eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:16:06.405590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266ff348-3119-4dda-bf81-ce96331e32eb service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:16:06.415048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 759/1530] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:16:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:16:08.585185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3a108-eaf0-4d0b-8680-566608b19bde None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:08.585721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3a108-eaf0-4d0b-8680-566608b19bde None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:08.585814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3a108-eaf0-4d0b-8680-566608b19bde None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:08.587129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 772/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:16:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:16:08.592250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a900cb11-98e9-45d6-804f-596b9f154177 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:08.592675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a900cb11-98e9-45d6-804f-596b9f154177 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:08.592806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a900cb11-98e9-45d6-804f-596b9f154177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:08.612356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a900cb11-98e9-45d6-804f-596b9f154177 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:16:08.613575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a900cb11-98e9-45d6-804f-596b9f154177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:08.621211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 760/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:16:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:16:08.908869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3a108-eaf0-4d0b-8680-566608b19bde None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:16:08.910589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cea3a108-eaf0-4d0b-8680-566608b19bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:08.923195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3a108-eaf0-4d0b-8680-566608b19bde None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:16:08.927221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd73a391-0afd-42e8-afd3-e1e4d74b4476 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:16:08.927814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd73a391-0afd-42e8-afd3-e1e4d74b4476 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:08.928095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd73a391-0afd-42e8-afd3-e1e4d74b4476 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:08.928367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd73a391-0afd-42e8-afd3-e1e4d74b4476 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:08.936970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd73a391-0afd-42e8-afd3-e1e4d74b4476 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:16:08.938451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd73a391-0afd-42e8-afd3-e1e4d74b4476 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:16:08.948092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 773/1533] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:16:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:16:12.642979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cebf36-32da-469e-8c1c-90dd76bb8226 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:12.643553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cebf36-32da-469e-8c1c-90dd76bb8226 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:12.643816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cebf36-32da-469e-8c1c-90dd76bb8226 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:12.645223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 761/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:16:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:16:12.650623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab27cb-2a89-4dda-b925-76277248f27e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:12.651058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab27cb-2a89-4dda-b925-76277248f27e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:12.651163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab27cb-2a89-4dda-b925-76277248f27e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:12.659535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=81 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:16:12.672657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-6dab27cb-2a89-4dda-b925-76277248f27e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:16:12.673900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6dab27cb-2a89-4dda-b925-76277248f27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:12.681723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 774/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:16:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:16:12.989058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cebf36-32da-469e-8c1c-90dd76bb8226 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:16:12.990724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e7cebf36-32da-469e-8c1c-90dd76bb8226 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:13.056825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:16:13.071154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cebf36-32da-469e-8c1c-90dd76bb8226 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:16:13.074712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb862125-bb26-4fa9-86a8-8517142c003c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:16:13.075321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb862125-bb26-4fa9-86a8-8517142c003c service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:13.075604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb862125-bb26-4fa9-86a8-8517142c003c service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:13.075882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb862125-bb26-4fa9-86a8-8517142c003c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:13.085572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb862125-bb26-4fa9-86a8-8517142c003c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:16:13.087002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb862125-bb26-4fa9-86a8-8517142c003c service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:16:13.096305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 762/1536] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:16:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:16:15.248817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15371617-cf57-44ac-adb2-24961f74e6e6 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:15.249380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15371617-cf57-44ac-adb2-24961f74e6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:15.249492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15371617-cf57-44ac-adb2-24961f74e6e6 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:15.250930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 775/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sat Oct 1 04:16:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:16:15.256308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ef6d0c-df7c-4b20-b48f-ae0942ab1cfc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:15.256766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ef6d0c-df7c-4b20-b48f-ae0942ab1cfc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:15.256876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ef6d0c-df7c-4b20-b48f-ae0942ab1cfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:15.275676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c1ef6d0c-df7c-4b20-b48f-ae0942ab1cfc None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:16:15.276857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c1ef6d0c-df7c-4b20-b48f-ae0942ab1cfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:15.285431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 763/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sat Oct 1 04:16:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:16:15.587813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15371617-cf57-44ac-adb2-24961f74e6e6 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:16:15.589543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-15371617-cf57-44ac-adb2-24961f74e6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:16:15.604675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15371617-cf57-44ac-adb2-24961f74e6e6 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:16:15.609488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20805ab7-e4ef-44b5-b72e-fe782338ca76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:16:15.610180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20805ab7-e4ef-44b5-b72e-fe782338ca76 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:16:15.610478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20805ab7-e4ef-44b5-b72e-fe782338ca76 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:16:15.610799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20805ab7-e4ef-44b5-b72e-fe782338ca76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:16:15.620448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20805ab7-e4ef-44b5-b72e-fe782338ca76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:16:15.621889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20805ab7-e4ef-44b5-b72e-fe782338ca76 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:16:15.631022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 776/1539] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:16:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:16:25.270984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=11 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:16:25.627779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=8 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:38.520712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1dd967-d391-4b93-82af-eb1b9b2f0768 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.521907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1dd967-d391-4b93-82af-eb1b9b2f0768 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.522949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1dd967-d391-4b93-82af-eb1b9b2f0768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.557763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cb1dd967-d391-4b93-82af-eb1b9b2f0768 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:38.559500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cb1dd967-d391-4b93-82af-eb1b9b2f0768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:38.575770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 764/1540] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:38.585927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c351ba8-26f6-4dca-96ec-a6bbd822d762 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.586537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c351ba8-26f6-4dca-96ec-a6bbd822d762 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.586537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c351ba8-26f6-4dca-96ec-a6bbd822d762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.622919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7c351ba8-26f6-4dca-96ec-a6bbd822d762 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:38.624243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7c351ba8-26f6-4dca-96ec-a6bbd822d762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:38.634094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 777/1541] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:38.639119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1dd967-d391-4b93-82af-eb1b9b2f0768 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:38.655586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1dd967-d391-4b93-82af-eb1b9b2f0768 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:38.667021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bd71fc-e14d-4651-ad86-4bbed190b4ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:38.668095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bd71fc-e14d-4651-ad86-4bbed190b4ff admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.668767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bd71fc-e14d-4651-ad86-4bbed190b4ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.669294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bd71fc-e14d-4651-ad86-4bbed190b4ff admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.670181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bd71fc-e14d-4651-ad86-4bbed190b4ff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:38.670718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bd71fc-e14d-4651-ad86-4bbed190b4ff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:38.672553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bd71fc-e14d-4651-ad86-4bbed190b4ff admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:38.691346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 765/1542] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:38.702870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c351ba8-26f6-4dca-96ec-a6bbd822d762 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:38.719024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c351ba8-26f6-4dca-96ec-a6bbd822d762 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:38.720563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df7e4-2929-4ccb-b3db-9d88c3b64dee None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.721322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df7e4-2929-4ccb-b3db-9d88c3b64dee None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.721827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df7e4-2929-4ccb-b3db-9d88c3b64dee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.724256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:38.724878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.725317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.725592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.728957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:38.728957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:38.748971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-087df7e4-2929-4ccb-b3db-9d88c3b64dee None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:38.750701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-087df7e4-2929-4ccb-b3db-9d88c3b64dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:38.767933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 766/1543] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:38.776703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d3cbb-8dd5-4df7-b655-1d57e2614aac None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.777579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d3cbb-8dd5-4df7-b655-1d57e2614aac None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.778208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d3cbb-8dd5-4df7-b655-1d57e2614aac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.790756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 778/1544] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/projects => generated 373 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:38.806445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:38.811520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-df8d3cbb-8dd5-4df7-b655-1d57e2614aac None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:38.812991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-df8d3cbb-8dd5-4df7-b655-1d57e2614aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:38.911143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 767/1545] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:38.915516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d3cbb-8dd5-4df7-b655-1d57e2614aac None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:38.919728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05906d4e-901c-483c-864f-ee82d1ceae35 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:38.935073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d3cbb-8dd5-4df7-b655-1d57e2614aac None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:38.941612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baaf8e6e-a6ae-465a-ad48-85b2288667b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:38.942015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f0bfff-2768-4280-a7d7-207fcc123297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:38.942648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f0bfff-2768-4280-a7d7-207fcc123297 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.942925 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f0bfff-2768-4280-a7d7-207fcc123297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.943236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf8e6e-a6ae-465a-ad48-85b2288667b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.943358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f0bfff-2768-4280-a7d7-207fcc123297 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.943831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf8e6e-a6ae-465a-ad48-85b2288667b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.943951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f0bfff-2768-4280-a7d7-207fcc123297 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:38.944424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaf8e6e-a6ae-465a-ad48-85b2288667b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.944973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING py.warnings [None req-62f0bfff-2768-4280-a7d7-207fcc123297 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:17:38.944973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: warnings.warn(msg) Oct 01 04:17:38.944973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: Oct 01 04:17:38.945711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaf8e6e-a6ae-465a-ad48-85b2288667b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:38.946199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f0bfff-2768-4280-a7d7-207fcc123297 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:38.946469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaf8e6e-a6ae-465a-ad48-85b2288667b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:38.948257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baaf8e6e-a6ae-465a-ad48-85b2288667b8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:38.977834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 768/1546] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:17:38.979539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446e90f-d05e-4f22-b4c1-326e0b959619 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.980045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446e90f-d05e-4f22-b4c1-326e0b959619 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.980544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446e90f-d05e-4f22-b4c1-326e0b959619 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:38.993554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 779/1547] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/users => generated 287 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:38.994935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c675efb-c3ee-48c7-91c7-f92307de8847 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:38.995140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c675efb-c3ee-48c7-91c7-f92307de8847 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:38.995416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c675efb-c3ee-48c7-91c7-f92307de8847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.012865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9446e90f-d05e-4f22-b4c1-326e0b959619 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.014468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9446e90f-d05e-4f22-b4c1-326e0b959619 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.028262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5c675efb-c3ee-48c7-91c7-f92307de8847 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.029547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5c675efb-c3ee-48c7-91c7-f92307de8847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.036966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 769/1548] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.037920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9446e90f-d05e-4f22-b4c1-326e0b959619 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.040269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 780/1549] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.041666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180a58f8-369c-4ea5-bbce-8a5a2549b85d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.042018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180a58f8-369c-4ea5-bbce-8a5a2549b85d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.042440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180a58f8-369c-4ea5-bbce-8a5a2549b85d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.056342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9446e90f-d05e-4f22-b4c1-326e0b959619 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.062375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09201ef2-4e22-4be4-8a33-7d36835f8620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.063125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09201ef2-4e22-4be4-8a33-7d36835f8620 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.063410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09201ef2-4e22-4be4-8a33-7d36835f8620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.064902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09201ef2-4e22-4be4-8a33-7d36835f8620 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.065615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09201ef2-4e22-4be4-8a33-7d36835f8620 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.066751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-09201ef2-4e22-4be4-8a33-7d36835f8620 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:17:39.066751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:17:39.066751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:17:39.067965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09201ef2-4e22-4be4-8a33-7d36835f8620 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.071784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-180a58f8-369c-4ea5-bbce-8a5a2549b85d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.073053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-180a58f8-369c-4ea5-bbce-8a5a2549b85d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.081808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 781/1550] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.082601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180a58f8-369c-4ea5-bbce-8a5a2549b85d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.093136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180a58f8-369c-4ea5-bbce-8a5a2549b85d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.098595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dbb7f8-8566-4729-84a0-b70015ce8a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.099200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dbb7f8-8566-4729-84a0-b70015ce8a80 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.099477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dbb7f8-8566-4729-84a0-b70015ce8a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.099746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dbb7f8-8566-4729-84a0-b70015ce8a80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.100182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 770/1551] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/projects => generated 363 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.100400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dbb7f8-8566-4729-84a0-b70015ce8a80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.102453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff86c84b-cab7-4124-af8d-51572c9e7f31 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.102453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff86c84b-cab7-4124-af8d-51572c9e7f31 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.102688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dbb7f8-8566-4729-84a0-b70015ce8a80 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.102780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff86c84b-cab7-4124-af8d-51572c9e7f31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.113511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 782/1552] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:39.116459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f89a5c-f5b1-4ebd-872e-a5769eec39c4 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.116459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f89a5c-f5b1-4ebd-872e-a5769eec39c4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.116459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f89a5c-f5b1-4ebd-872e-a5769eec39c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.121105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ff86c84b-cab7-4124-af8d-51572c9e7f31 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.122325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ff86c84b-cab7-4124-af8d-51572c9e7f31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.134958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-70f89a5c-f5b1-4ebd-872e-a5769eec39c4 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.136145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-70f89a5c-f5b1-4ebd-872e-a5769eec39c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.183510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 771/1553] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.184760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8848542-5c77-4e01-baaf-f870423a947b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.185088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8848542-5c77-4e01-baaf-f870423a947b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.185366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8848542-5c77-4e01-baaf-f870423a947b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.198422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 783/1554] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.199764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1936244-e409-42ca-a016-7b6d8f13466c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.200056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1936244-e409-42ca-a016-7b6d8f13466c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.200332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1936244-e409-42ca-a016-7b6d8f13466c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.207760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d8848542-5c77-4e01-baaf-f870423a947b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.209006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d8848542-5c77-4e01-baaf-f870423a947b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.216855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 772/1555] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.218130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0562a8-8461-42a3-806d-77a7dba0594a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.218414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0562a8-8461-42a3-806d-77a7dba0594a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.218689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0562a8-8461-42a3-806d-77a7dba0594a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.224350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b1936244-e409-42ca-a016-7b6d8f13466c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.225578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b1936244-e409-42ca-a016-7b6d8f13466c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.233462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 784/1556] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.234703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c81151d-d7c3-4698-b886-3b1b1428756e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.234984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c81151d-d7c3-4698-b886-3b1b1428756e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.235251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c81151d-d7c3-4698-b886-3b1b1428756e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.238754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-db0562a8-8461-42a3-806d-77a7dba0594a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.239939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-db0562a8-8461-42a3-806d-77a7dba0594a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.248004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 773/1557] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.248766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0562a8-8461-42a3-806d-77a7dba0594a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.255996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7c81151d-d7c3-4698-b886-3b1b1428756e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.257303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7c81151d-d7c3-4698-b886-3b1b1428756e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.261263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0562a8-8461-42a3-806d-77a7dba0594a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.265086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 785/1558] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.265802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b888dea5-b3e6-43af-a4b2-f70c5e20fdf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.265951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c81151d-d7c3-4698-b886-3b1b1428756e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.266469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888dea5-b3e6-43af-a4b2-f70c5e20fdf2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.266753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888dea5-b3e6-43af-a4b2-f70c5e20fdf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.268071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888dea5-b3e6-43af-a4b2-f70c5e20fdf2 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.268411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b888dea5-b3e6-43af-a4b2-f70c5e20fdf2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.270383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b888dea5-b3e6-43af-a4b2-f70c5e20fdf2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.278024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c81151d-d7c3-4698-b886-3b1b1428756e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.283159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.283750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.284044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.284337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.284943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.286697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.296132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 774/1559] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/users => generated 282 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.296132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afb872f-4a89-45dd-b33b-f95e1fb6c2be None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.296132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afb872f-4a89-45dd-b33b-f95e1fb6c2be None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.296132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afb872f-4a89-45dd-b33b-f95e1fb6c2be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.297317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 786/1560] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:39.298687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.303484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:39.307091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:39.311359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb28393-9d5d-4507-88c7-55b9862e7325 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.318023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.318614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.318888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.319155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.319383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3afb872f-4a89-45dd-b33b-f95e1fb6c2be None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.319785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:39.320089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.320584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3afb872f-4a89-45dd-b33b-f95e1fb6c2be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.321104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING py.warnings [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:17:39.321104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: warnings.warn(msg) Oct 01 04:17:39.321104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: Oct 01 04:17:39.321992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8115b6e8-9906-449e-ac38-7cc2b867be1a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.328437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 775/1561] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.329239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afb872f-4a89-45dd-b33b-f95e1fb6c2be None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.344900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 787/1562] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:39.345216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afb872f-4a89-45dd-b33b-f95e1fb6c2be None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.345382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53014a5-9e22-442f-a520-dfb3d46f75d7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.345382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53014a5-9e22-442f-a520-dfb3d46f75d7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.345382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53014a5-9e22-442f-a520-dfb3d46f75d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.345667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3a8a25-fa60-46f8-babc-80d85445d261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.345667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a8a25-fa60-46f8-babc-80d85445d261 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.345667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a8a25-fa60-46f8-babc-80d85445d261 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.345667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a8a25-fa60-46f8-babc-80d85445d261 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.346356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3a8a25-fa60-46f8-babc-80d85445d261 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:39.346670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3a8a25-fa60-46f8-babc-80d85445d261 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.348260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3a8a25-fa60-46f8-babc-80d85445d261 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.359559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a53014a5-9e22-442f-a520-dfb3d46f75d7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.359708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 776/1563] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:39.361367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a53014a5-9e22-442f-a520-dfb3d46f75d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.361499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e9be0-0625-4b0e-835a-aed3e26e2771 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.362277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e9be0-0625-4b0e-835a-aed3e26e2771 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.362559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e9be0-0625-4b0e-835a-aed3e26e2771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.376030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 788/1564] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.377338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53014a5-9e22-442f-a520-dfb3d46f75d7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.386403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-709e9be0-0625-4b0e-835a-aed3e26e2771 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.386884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53014a5-9e22-442f-a520-dfb3d46f75d7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.387628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-709e9be0-0625-4b0e-835a-aed3e26e2771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.391354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.391977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.392263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.392528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.393191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:39.393490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.395058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.396298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 777/1565] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.400471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e9be0-0625-4b0e-835a-aed3e26e2771 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.412252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 789/1566] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:39.412252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.414760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e9be0-0625-4b0e-835a-aed3e26e2771 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.418835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a02363-e8eb-4953-b602-5cab19a71579 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.419424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.420021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.420336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.420626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.421320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.422905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.423885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.424630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.424906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.425202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] PATH_INFO: `/v3/projects/3ebfba30505f48c181cb04960d38aa63/users/d7e1763c88334c29a14bb192bfe32bb2/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.434099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 778/1567] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:39.434959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.440988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ebfba30505f48c181cb04960d38aa63, user_id=d7e1763c88334c29a14bb192bfe32bb2, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.443818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.445862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11950c-3632-4ebe-b82c-0818c3a1ba43 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.451913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.453031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.453682 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.454301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.455292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:39.455778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.458574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.461750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 790/1568] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:39 2022] PUT /identity/v3/projects/3ebfba30505f48c181cb04960d38aa63/users/d7e1763c88334c29a14bb192bfe32bb2/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:39.463227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.487533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 779/1569] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:39.488359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.497853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3a0b21-a9d6-4e0d-98e8-c6596ae0ea72 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.540207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a63e39-b632-413c-beac-44c90a661d06 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.544797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.545445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.545729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.546007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.546793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.549175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.565833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.566508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.566793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.567072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.567756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.569744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.579198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 791/1570] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/projects => generated 389 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.579198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.600545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 780/1571] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/projects => generated 371 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.601845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.639761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876424b2-da9e-4348-952e-b2a0c00aa55e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.667432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4de7a4-061c-4831-8ab2-04a2c6837c71 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.672268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15814594-9782-4b29-a819-9d011183105e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.672879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15814594-9782-4b29-a819-9d011183105e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.673818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15814594-9782-4b29-a819-9d011183105e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.674465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15814594-9782-4b29-a819-9d011183105e admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.675044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15814594-9782-4b29-a819-9d011183105e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:39.675350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15814594-9782-4b29-a819-9d011183105e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.677000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15814594-9782-4b29-a819-9d011183105e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.678925 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:39.689672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 781/1572] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:39.690479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15814594-9782-4b29-a819-9d011183105e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.699368 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.699982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.700273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.700550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.701245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:39.701832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.702099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15814594-9782-4b29-a819-9d011183105e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.703426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.707323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8489d5e0-1845-4d17-ab96-f6031f828dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.707914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8489d5e0-1845-4d17-ab96-f6031f828dc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.708246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8489d5e0-1845-4d17-ab96-f6031f828dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.708519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8489d5e0-1845-4d17-ab96-f6031f828dc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.709216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8489d5e0-1845-4d17-ab96-f6031f828dc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.711082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8489d5e0-1845-4d17-ab96-f6031f828dc8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.714808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 792/1573] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:39.720998 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.726563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350542df-99df-4d3d-9f1b-7af26e04b630 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.731306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.731907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.732190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.732468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.733143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.734752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.744891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 782/1574] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.746259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad71ff-7a8e-463d-a241-9654069d4f15 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.746605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad71ff-7a8e-463d-a241-9654069d4f15 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.746941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad71ff-7a8e-463d-a241-9654069d4f15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.749065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 793/1575] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:39.749065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.760667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec2db7-7835-41f1-acb9-d4ec360cf91d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.766478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:39.776797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-57ad71ff-7a8e-463d-a241-9654069d4f15 None None] MFA Rules not processed for user `d7e1763c88334c29a14bb192bfe32bb2`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:39.778395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-57ad71ff-7a8e-463d-a241-9654069d4f15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:39.820649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.821295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.821574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.821843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.822470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.824380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.854397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 794/1576] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/projects => generated 373 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.855258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.860408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 783/1577] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:39.865657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad71ff-7a8e-463d-a241-9654069d4f15 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.876672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad71ff-7a8e-463d-a241-9654069d4f15 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.922640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c69619-86bc-469b-a655-eacf3df4ef67 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.928028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.928664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.928963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.929243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.929869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.931822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.939987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.940542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.940829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.941138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.941762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.943685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.952539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 795/1578] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/users => generated 295 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.953375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.963922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02864fb7-9eb5-4255-ab89-bfcbcca891e2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.968238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 784/1579] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/users => generated 286 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:39.972166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.972297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:39.972390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.972390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.972390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.972390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.973057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:39.978786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe532e82-59f3-40f9-a34a-5a1a330a1a03 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:39.983502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:39.983787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:39.984063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:39.984333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:39.985160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:39.986964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.000697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 796/1580] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/projects => generated 389 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:40.002445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.015629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 785/1581] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:39 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:40.016971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.021267 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:40.068414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fb7cd5-8a7d-41a5-9ddd-ccb74ad7b242 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.078757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f229e-1086-4880-9225-3e3d95f24a34 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.084408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.085111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.085852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.086131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] PATH_INFO: `/v3/projects/f4fe4437a4ee4d8f86d3834942b1fe3d/users/9e394c96cb604d0d8a4f53a8403d53ac/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.099741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:40.104153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4fe4437a4ee4d8f86d3834942b1fe3d, user_id=9e394c96cb604d0d8a4f53a8403d53ac, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.105667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:17:40.105667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:17:40.105667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:17:40.107758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.122253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 786/1582] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:40 2022] PUT /identity/v3/projects/f4fe4437a4ee4d8f86d3834942b1fe3d/users/9e394c96cb604d0d8a4f53a8403d53ac/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:40.123037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.127444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44efcbca-fe5d-480a-ab8a-06f2c0048bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.128162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44efcbca-fe5d-480a-ab8a-06f2c0048bf3 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.129027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44efcbca-fe5d-480a-ab8a-06f2c0048bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.129027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44efcbca-fe5d-480a-ab8a-06f2c0048bf3 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.129606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44efcbca-fe5d-480a-ab8a-06f2c0048bf3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.131729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44efcbca-fe5d-480a-ab8a-06f2c0048bf3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.160353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 797/1583] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/users => generated 287 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:40.160353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d0743e-2266-4de2-98d2-d6f6bb601db3 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.160353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d0743e-2266-4de2-98d2-d6f6bb601db3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.160353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d0743e-2266-4de2-98d2-d6f6bb601db3 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.160832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 798/1584] 10.222.0.26 () {54 vars in 961 bytes} [Sat Oct 1 04:17:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:17:40.161233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d0743e-2266-4de2-98d2-d6f6bb601db3 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.191991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2f41bc-80d8-4f34-b5a4-eee8e617324d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.199255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.199865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.200142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.200413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.201176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.203095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.225393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 787/1585] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:40.228787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.231549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d0743e-2266-4de2-98d2-d6f6bb601db3 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.236217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.236825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.236981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfe2ff0-39fe-4164-9e9a-2d40c30a1f6c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.237160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.237436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.238078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.239672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.241490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.242438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.242716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.242985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.243616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.245659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.249733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 799/1586] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.250592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.257873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 788/1587] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.258525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.260784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71b842b-8957-49e1-bbcd-0af4d7dc08d0 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.265371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6b2fa0-27eb-4e8d-a8a1-67f9398a4d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.265959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6b2fa0-27eb-4e8d-a8a1-67f9398a4d06 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.266235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6b2fa0-27eb-4e8d-a8a1-67f9398a4d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.266509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6b2fa0-27eb-4e8d-a8a1-67f9398a4d06 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.267675 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6b2fa0-27eb-4e8d-a8a1-67f9398a4d06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.268252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d48fbd-1a83-42b6-a722-61d667196b64 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.269731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6b2fa0-27eb-4e8d-a8a1-67f9398a4d06 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.272737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.273341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.273623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.273891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.274500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.276368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.293922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 800/1588] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/users => generated 295 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:40.295128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc0a72b-4339-4c20-b378-244a0c42fa00 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.295418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc0a72b-4339-4c20-b378-244a0c42fa00 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.295691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc0a72b-4339-4c20-b378-244a0c42fa00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.300269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 789/1589] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/users => generated 289 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:40.300995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.312165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03929c2-b998-4f19-b1ee-0c02650db64a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.316688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e920e2-cc89-4cf5-ab37-98518459c801 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.317309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e920e2-cc89-4cf5-ab37-98518459c801 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.317589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e920e2-cc89-4cf5-ab37-98518459c801 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.317859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e920e2-cc89-4cf5-ab37-98518459c801 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.318492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e920e2-cc89-4cf5-ab37-98518459c801 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.320050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e920e2-cc89-4cf5-ab37-98518459c801 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.327285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-acc0a72b-4339-4c20-b378-244a0c42fa00 None None] MFA Rules not processed for user `9e394c96cb604d0d8a4f53a8403d53ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:40.328553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-acc0a72b-4339-4c20-b378-244a0c42fa00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:40.330296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 790/1590] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.331395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327b81c-6a2a-496c-a7fa-7fe19e1d3459 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.331680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327b81c-6a2a-496c-a7fa-7fe19e1d3459 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.331953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327b81c-6a2a-496c-a7fa-7fe19e1d3459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.347899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:40.371083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6327b81c-6a2a-496c-a7fa-7fe19e1d3459 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:40.372331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6327b81c-6a2a-496c-a7fa-7fe19e1d3459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:40.394325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 801/1591] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:40.395249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc0a72b-4339-4c20-b378-244a0c42fa00 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.405569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc0a72b-4339-4c20-b378-244a0c42fa00 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.410208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.410811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.411089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.411359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.411995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.413611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.423574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 802/1592] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.424463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.434400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5585dd2c-af72-4337-9d2e-5718284cae23 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.439395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.440136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.440415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.440683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.441332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.442871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.443697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 791/1593] 10.222.0.26 () {58 vars in 1058 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:40.449225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6327b81c-6a2a-496c-a7fa-7fe19e1d3459 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.452624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 803/1594] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.453446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.460544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6327b81c-6a2a-496c-a7fa-7fe19e1d3459 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.464258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c368d1-e8bc-499b-902e-c7ddcca4998a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.465072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.465674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.465953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.466228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.466865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.468460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.468820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.469419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.469704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.470543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.471204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.472750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.478783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 792/1595] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.479656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.488218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 804/1596] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.488218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.489324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8024b1-7800-4de8-86ba-f61d07aed21d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.494549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.495141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.495545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.495816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.496470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.497310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf10a84-c51b-4abf-a53f-81d736ef91da admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.498098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.499439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:40.505598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.505598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.505598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.505598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.505992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.507095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.510154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 793/1597] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.511692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.522425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 805/1598] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.522425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.524792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efe87c5-4fa0-4cfe-84f5-13ddf019e6ac admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.529701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.530253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba35f5f4-7481-4c7d-9919-b124548eba4f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.530422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.530826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.531106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.531738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.534240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.535499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.536090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.536369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.536648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.537315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.538888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.549578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 806/1599] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.550383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.566342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 794/1600] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:40.566342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.566977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e0152c-bf26-46dc-8545-70851e0e83f0 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.624739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2bba-79a8-42ed-bbf0-967b00d75ce1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.628975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.629588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.629872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.630075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.630194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.630751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.631057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.631301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] PATH_INFO: `/v3/projects/839c91c5262a49768c3e937b5a034137/users/7014a1defd81461787318bc27d56b9aa/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.646550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] RBAC: Authorizing `identity:create_grant(project_id=839c91c5262a49768c3e937b5a034137, user_id=7014a1defd81461787318bc27d56b9aa, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.649342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.663643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 795/1601] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:40 2022] PUT /identity/v3/projects/839c91c5262a49768c3e937b5a034137/users/7014a1defd81461787318bc27d56b9aa/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:40.664430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.690533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.692120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.700982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 807/1602] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:40.701820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.706620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:40.712264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e54569-a03b-4c7f-962c-eac5f29f8a05 service nova] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.729133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28efb08-09da-470e-a44e-bd37c4ae50de admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.734763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.736414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.736886 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.737258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] PATH_INFO: `/v3/projects/d741cb6333b846708b3d55abd77b3934/users/bbe1637bd0df445b87f04561ccbbaafe/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.752724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d741cb6333b846708b3d55abd77b3934, user_id=bbe1637bd0df445b87f04561ccbbaafe, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.755662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.780013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 796/1603] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:40 2022] PUT /identity/v3/projects/d741cb6333b846708b3d55abd77b3934/users/bbe1637bd0df445b87f04561ccbbaafe/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:40.780013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.781514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.782353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.783375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.783957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.784454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.786118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.799743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 808/1604] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.799743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.808806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc87a5c-73cb-480d-9f89-d20292d30e5e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.831304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:40.856538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1850c60a-8b19-4b44-b823-a3d5bd7d9c70 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.862098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.862666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.862951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.863230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.863874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.865752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.871192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.871925 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.872209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.872484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] PATH_INFO: `/v3/projects/1d594ba574fb4f8b8b97d4e3925b0d32/users/791c81198cc74fac808d2b40209d7a32/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.877638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 797/1605] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:40 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:40.878608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.887577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d594ba574fb4f8b8b97d4e3925b0d32, user_id=791c81198cc74fac808d2b40209d7a32, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.890271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.890810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6328c7b0-12a2-49d0-89ef-bd695d47151d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:40.897318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd236f80-b991-4b26-87aa-7eabe83f4a12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:40.898065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd236f80-b991-4b26-87aa-7eabe83f4a12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.898548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd236f80-b991-4b26-87aa-7eabe83f4a12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.898896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd236f80-b991-4b26-87aa-7eabe83f4a12 admin admin] PATH_INFO: `/v3/projects/16cd7bcb315d4dc58f73a9918adb57e2/users/3caf1cc366434d418c474d7a1879faa7/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.907111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 809/1606] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:40 2022] PUT /identity/v3/projects/1d594ba574fb4f8b8b97d4e3925b0d32/users/791c81198cc74fac808d2b40209d7a32/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:40.908387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:40.915851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd236f80-b991-4b26-87aa-7eabe83f4a12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16cd7bcb315d4dc58f73a9918adb57e2, user_id=3caf1cc366434d418c474d7a1879faa7, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:40.919079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd236f80-b991-4b26-87aa-7eabe83f4a12 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:40.938584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 798/1607] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:40 2022] PUT /identity/v3/projects/16cd7bcb315d4dc58f73a9918adb57e2/users/3caf1cc366434d418c474d7a1879faa7/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:40.940069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f45c73-2d89-4c6e-a07d-b40e7d2f6891 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:40.940358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f45c73-2d89-4c6e-a07d-b40e7d2f6891 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:40.940624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f45c73-2d89-4c6e-a07d-b40e7d2f6891 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:40.961040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-54f45c73-2d89-4c6e-a07d-b40e7d2f6891 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:40.962236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-54f45c73-2d89-4c6e-a07d-b40e7d2f6891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:40.981515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2615beeb-d5b5-4efd-a7b1-42ef88419224 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.013749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:41.024973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 799/1608] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:41.026162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67397d8b-10a2-49d2-be62-957d8faf72ab None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.026447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67397d8b-10a2-49d2-be62-957d8faf72ab None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.026722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67397d8b-10a2-49d2-be62-957d8faf72ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.054684 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.055417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.055698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.056005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.057320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.058693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.064122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-67397d8b-10a2-49d2-be62-957d8faf72ab None None] MFA Rules not processed for user `bbe1637bd0df445b87f04561ccbbaafe`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:41.065355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-67397d8b-10a2-49d2-be62-957d8faf72ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:41.086349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 810/1609] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:40 2022] POST /identity/v3/users => generated 289 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:41.086349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.095429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1c4677-8338-407c-89b0-be8eaeb85500 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.101099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352ee584-b3ac-4d0d-bc1d-9ff22bf4f694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.101857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352ee584-b3ac-4d0d-bc1d-9ff22bf4f694 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.102136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352ee584-b3ac-4d0d-bc1d-9ff22bf4f694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.103065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352ee584-b3ac-4d0d-bc1d-9ff22bf4f694 admin admin] PATH_INFO: `/v3/projects/19d87f6d9adf4ce98e934187ec8ae0d7/users/ccd631925a0a4f5ca8be5d3915778487/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.119599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352ee584-b3ac-4d0d-bc1d-9ff22bf4f694 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19d87f6d9adf4ce98e934187ec8ae0d7, user_id=ccd631925a0a4f5ca8be5d3915778487, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.122810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352ee584-b3ac-4d0d-bc1d-9ff22bf4f694 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.135409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 800/1610] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:41.139138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 811/1611] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:41 2022] PUT /identity/v3/projects/19d87f6d9adf4ce98e934187ec8ae0d7/users/ccd631925a0a4f5ca8be5d3915778487/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:41.140447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a3390c-cbd0-4c35-a8cd-e45869e5f352 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.140736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a3390c-cbd0-4c35-a8cd-e45869e5f352 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.141073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a3390c-cbd0-4c35-a8cd-e45869e5f352 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.142470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67397d8b-10a2-49d2-be62-957d8faf72ab None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.158956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67397d8b-10a2-49d2-be62-957d8faf72ab None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.175532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a6a3390c-cbd0-4c35-a8cd-e45869e5f352 None None] MFA Rules not processed for user `791c81198cc74fac808d2b40209d7a32`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:41.176783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a6a3390c-cbd0-4c35-a8cd-e45869e5f352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:41.205687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:41.225616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b86c2a-9015-4e2a-855e-2a988fe28a5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.226372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b86c2a-9015-4e2a-855e-2a988fe28a5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.226720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b86c2a-9015-4e2a-855e-2a988fe28a5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.227065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b86c2a-9015-4e2a-855e-2a988fe28a5c admin admin] PATH_INFO: `/v3/projects/c1b321c355e94b8496955f09f547f561/users/bfcc0e7f557e4334a808f5dfaffe93db/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.244731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b86c2a-9015-4e2a-855e-2a988fe28a5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1b321c355e94b8496955f09f547f561, user_id=bfcc0e7f557e4334a808f5dfaffe93db, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.251987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b86c2a-9015-4e2a-855e-2a988fe28a5c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.269082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 801/1612] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:41 2022] PUT /identity/v3/projects/c1b321c355e94b8496955f09f547f561/users/bfcc0e7f557e4334a808f5dfaffe93db/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:41.274223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead5b68-45b2-44a2-a685-cf2fb4e5a252 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.274507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead5b68-45b2-44a2-a685-cf2fb4e5a252 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.274777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead5b68-45b2-44a2-a685-cf2fb4e5a252 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.276650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 812/1613] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:41.277848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a3390c-cbd0-4c35-a8cd-e45869e5f352 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.294541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-eead5b68-45b2-44a2-a685-cf2fb4e5a252 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:41.295775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-eead5b68-45b2-44a2-a685-cf2fb4e5a252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:41.333966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a3390c-cbd0-4c35-a8cd-e45869e5f352 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.356761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 802/1614] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:41.357647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eead5b68-45b2-44a2-a685-cf2fb4e5a252 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.368787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eead5b68-45b2-44a2-a685-cf2fb4e5a252 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.374260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73df0ea1-f730-4787-907d-5b7f1d432c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.374425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:41.374880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73df0ea1-f730-4787-907d-5b7f1d432c1e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.375178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73df0ea1-f730-4787-907d-5b7f1d432c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.375436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73df0ea1-f730-4787-907d-5b7f1d432c1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.376091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73df0ea1-f730-4787-907d-5b7f1d432c1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.378411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73df0ea1-f730-4787-907d-5b7f1d432c1e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.387482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.388080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.388360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.388462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 803/1615] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:41.388631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.389757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a09786-6197-4867-a32a-31c3b14660bf None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.390046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a09786-6197-4867-a32a-31c3b14660bf None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.390323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a09786-6197-4867-a32a-31c3b14660bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.400755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.402503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.410411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-82a09786-6197-4867-a32a-31c3b14660bf None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:41.411630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-82a09786-6197-4867-a32a-31c3b14660bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:41.412590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 813/1616] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:41.413225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.419549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 804/1617] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:41.425737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4131b2-3c34-4022-8f91-ecf184add10c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.426024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4131b2-3c34-4022-8f91-ecf184add10c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.426301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4131b2-3c34-4022-8f91-ecf184add10c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.432071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 805/1618] 10.222.0.26 () {54 vars in 961 bytes} [Sat Oct 1 04:17:41 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:17:41.433325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78955013-e6c6-4e66-b229-8208aea29f3c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.433610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78955013-e6c6-4e66-b229-8208aea29f3c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.433885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78955013-e6c6-4e66-b229-8208aea29f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.453509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-78955013-e6c6-4e66-b229-8208aea29f3c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:41.454914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-78955013-e6c6-4e66-b229-8208aea29f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:41.463617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 806/1619] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:41.472876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78955013-e6c6-4e66-b229-8208aea29f3c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.483390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78955013-e6c6-4e66-b229-8208aea29f3c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.489579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.490193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.490552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.490794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.491506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.492222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adac7fc1-0988-40c2-8ca6-4d1053eb5899 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.493488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.498071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.498808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.499185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.499563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.528979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 807/1620] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/projects => generated 373 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:41.529944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.580329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.582023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.592731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 814/1621] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:41.593597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.602017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae6dd5f-6696-49fa-ae17-0d7cc663b220 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.608228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc942c9-e0db-4578-a24a-0534af2cec73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.609016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc942c9-e0db-4578-a24a-0534af2cec73 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.609295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc942c9-e0db-4578-a24a-0534af2cec73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.609564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc942c9-e0db-4578-a24a-0534af2cec73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.610332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237b58b-b1f4-4123-8229-731fea5c46ca service nova] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.658923 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:41.690247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.690966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.691537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.691537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] PATH_INFO: `/v3/projects/9b206eb3f4e54053850c74d85123bb17/users/9690db63db3f4423bcdb7076008169ab/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.694194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc942c9-e0db-4578-a24a-0534af2cec73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.695751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc942c9-e0db-4578-a24a-0534af2cec73 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.706723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 808/1622] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:41.707537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b206eb3f4e54053850c74d85123bb17, user_id=9690db63db3f4423bcdb7076008169ab, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.708156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd25c16-763a-46f7-8e1d-a55a34d91d50 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.708503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd25c16-763a-46f7-8e1d-a55a34d91d50 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.708841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd25c16-763a-46f7-8e1d-a55a34d91d50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.710264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.724852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 815/1623] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:41 2022] PUT /identity/v3/projects/9b206eb3f4e54053850c74d85123bb17/users/9690db63db3f4423bcdb7076008169ab/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:41.725713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.728040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6cd25c16-763a-46f7-8e1d-a55a34d91d50 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:41.729476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6cd25c16-763a-46f7-8e1d-a55a34d91d50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:41.732299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:41.810732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c14541-ab7a-477d-864c-35a3bdd55340 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.814645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.815278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.815844 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.815971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.852327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 809/1624] 10.222.0.26 () {58 vars in 1058 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:41.854901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd25c16-763a-46f7-8e1d-a55a34d91d50 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.865489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd25c16-763a-46f7-8e1d-a55a34d91d50 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.869478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d193cfd-53a0-4656-bc2e-0b0546b2f090 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.869878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d193cfd-53a0-4656-bc2e-0b0546b2f090 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.870168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d193cfd-53a0-4656-bc2e-0b0546b2f090 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.870599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d193cfd-53a0-4656-bc2e-0b0546b2f090 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.902288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.904231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.918062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 816/1625] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:41.918810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.929636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eff3e83-989a-4e38-90ac-eacabc4f4b42 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.935981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e164bb-0f09-482d-a8b3-7e199243bb88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:41.936450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e164bb-0f09-482d-a8b3-7e199243bb88 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.936739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e164bb-0f09-482d-a8b3-7e199243bb88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.937371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e164bb-0f09-482d-a8b3-7e199243bb88 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.938163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e164bb-0f09-482d-a8b3-7e199243bb88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.941207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e164bb-0f09-482d-a8b3-7e199243bb88 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.948506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d193cfd-53a0-4656-bc2e-0b0546b2f090 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:41.950773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d193cfd-53a0-4656-bc2e-0b0546b2f090 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:41.966943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 810/1626] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:41.968261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ca5ff-0947-4a98-9d50-2dad4456f156 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.968588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ca5ff-0947-4a98-9d50-2dad4456f156 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.969117 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ca5ff-0947-4a98-9d50-2dad4456f156 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.976086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 811/1627] 10.222.0.26 () {54 vars in 912 bytes} [Sat Oct 1 04:17:41 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:17:41.977030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 817/1628] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:41 2022] POST /identity/v3/users => generated 293 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:41.977996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ca5ff-0947-4a98-9d50-2dad4456f156 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.978654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570cce41-93b3-4eb7-9f6c-844e21018a29 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:41.979099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570cce41-93b3-4eb7-9f6c-844e21018a29 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:41.979436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570cce41-93b3-4eb7-9f6c-844e21018a29 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:41.981690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 818/1629] 10.222.0.26 () {54 vars in 912 bytes} [Sat Oct 1 04:17:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:17:41.982418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570cce41-93b3-4eb7-9f6c-844e21018a29 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:41.993225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ca5ff-0947-4a98-9d50-2dad4456f156 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:41.999364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05a9582-4975-4c25-8dea-368b5946e7fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.000083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a9582-4975-4c25-8dea-368b5946e7fb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.000352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570cce41-93b3-4eb7-9f6c-844e21018a29 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.000486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a9582-4975-4c25-8dea-368b5946e7fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.000734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05a9582-4975-4c25-8dea-368b5946e7fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.002418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05a9582-4975-4c25-8dea-368b5946e7fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.004091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05a9582-4975-4c25-8dea-368b5946e7fb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.008005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7b29e-6ecf-45b0-93a5-dd60fbcf9f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.008629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7b29e-6ecf-45b0-93a5-dd60fbcf9f91 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.008918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7b29e-6ecf-45b0-93a5-dd60fbcf9f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.009249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7b29e-6ecf-45b0-93a5-dd60fbcf9f91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.010106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7b29e-6ecf-45b0-93a5-dd60fbcf9f91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.011885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7b29e-6ecf-45b0-93a5-dd60fbcf9f91 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.025858 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 812/1630] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:42.027635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7555866-0761-4d30-9da2-ac0dccf45bb4 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.027955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7555866-0761-4d30-9da2-ac0dccf45bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.028331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7555866-0761-4d30-9da2-ac0dccf45bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.045662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 819/1631] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:41 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:42.048170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579c47f8-8f5d-45f5-b772-1b9071ca2428 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.048699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579c47f8-8f5d-45f5-b772-1b9071ca2428 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.049369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579c47f8-8f5d-45f5-b772-1b9071ca2428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.052616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f7555866-0761-4d30-9da2-ac0dccf45bb4 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:42.054328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f7555866-0761-4d30-9da2-ac0dccf45bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:42.072438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 813/1632] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:17:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:42.073411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7555866-0761-4d30-9da2-ac0dccf45bb4 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:42.075712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-579c47f8-8f5d-45f5-b772-1b9071ca2428 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:42.077315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-579c47f8-8f5d-45f5-b772-1b9071ca2428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:42.085862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7555866-0761-4d30-9da2-ac0dccf45bb4 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.090024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 820/1633] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:17:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:42.090976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579c47f8-8f5d-45f5-b772-1b9071ca2428 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:42.091785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.092470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.092849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.093175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] PATH_INFO: `/v3/projects/9b206eb3f4e54053850c74d85123bb17/users/9690db63db3f4423bcdb7076008169ab/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.095812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b206eb3f4e54053850c74d85123bb17, user_id=9690db63db3f4423bcdb7076008169ab, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.099796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.105853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579c47f8-8f5d-45f5-b772-1b9071ca2428 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.122850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 814/1634] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:42 2022] PUT /identity/v3/projects/9b206eb3f4e54053850c74d85123bb17/users/9690db63db3f4423bcdb7076008169ab/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:42.124639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:42.169570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:42.198253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.199000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.199286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.200232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] PATH_INFO: `/v3/projects/313646c210bd4deb80ce02228c46a29a/users/92e78603d5d64f93847da5de8670ed5b/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.214260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f57c1a6-70d8-45c0-8302-4d2dace5616b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.219406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f663ba-cb4b-4116-bf51-8133ba2e9a25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.220446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f663ba-cb4b-4116-bf51-8133ba2e9a25 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.220446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f663ba-cb4b-4116-bf51-8133ba2e9a25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.220757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f663ba-cb4b-4116-bf51-8133ba2e9a25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.224440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] RBAC: Authorizing `identity:create_grant(project_id=313646c210bd4deb80ce02228c46a29a, user_id=92e78603d5d64f93847da5de8670ed5b, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.228195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.253247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:42.263692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 821/1635] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:42 2022] PUT /identity/v3/projects/313646c210bd4deb80ce02228c46a29a/users/92e78603d5d64f93847da5de8670ed5b/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:42.265208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:42.313828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f663ba-cb4b-4116-bf51-8133ba2e9a25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.316905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f663ba-cb4b-4116-bf51-8133ba2e9a25 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.333704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 815/1636] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:42.335296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1211e93-5f59-49bf-a7c6-735898f1f05a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.335658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1211e93-5f59-49bf-a7c6-735898f1f05a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.335991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1211e93-5f59-49bf-a7c6-735898f1f05a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.394674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b1211e93-5f59-49bf-a7c6-735898f1f05a None None] MFA Rules not processed for user `9690db63db3f4423bcdb7076008169ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:42.397302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b1211e93-5f59-49bf-a7c6-735898f1f05a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:42.412854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6198854d-1350-4b71-abaa-f93b37559741 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.422208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.423728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.426081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.426552 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.494669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 816/1637] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:42 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:42.500494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1211e93-5f59-49bf-a7c6-735898f1f05a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:42.598025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:42.618345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1211e93-5f59-49bf-a7c6-735898f1f05a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.625080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.626262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.626749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.627097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.629029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.630742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.638205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.639738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.650029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 817/1638] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:42 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:42.660594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:42.675010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 822/1639] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:42.695036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb81b21-7329-4150-aac0-f1cb0530bc4c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.709184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f6fd3-146e-4b5c-87b2-6b7f2b47abb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.710373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f6fd3-146e-4b5c-87b2-6b7f2b47abb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.712288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f6fd3-146e-4b5c-87b2-6b7f2b47abb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.712722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f6fd3-146e-4b5c-87b2-6b7f2b47abb5 admin admin] PATH_INFO: `/v3/projects/313646c210bd4deb80ce02228c46a29a/users/92e78603d5d64f93847da5de8670ed5b/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.717746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f6fd3-146e-4b5c-87b2-6b7f2b47abb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=313646c210bd4deb80ce02228c46a29a, user_id=92e78603d5d64f93847da5de8670ed5b, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:42.732758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f6fd3-146e-4b5c-87b2-6b7f2b47abb5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:42.752775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:42.763373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299c5489-1cd7-4328-b3d2-8b7d45c0dc23 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:42.765278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 818/1640] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:42 2022] PUT /identity/v3/projects/313646c210bd4deb80ce02228c46a29a/users/92e78603d5d64f93847da5de8670ed5b/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:42.783262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd91544-ab52-4f78-be5f-d3e0657df4bb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.783632 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd91544-ab52-4f78-be5f-d3e0657df4bb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.784191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd91544-ab52-4f78-be5f-d3e0657df4bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.850603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fdd91544-ab52-4f78-be5f-d3e0657df4bb None None] MFA Rules not processed for user `92e78603d5d64f93847da5de8670ed5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:42.853155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fdd91544-ab52-4f78-be5f-d3e0657df4bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:42.870527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:42.872371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:42.872647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:42.872916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:42.987048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 819/1641] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:42 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:42.990958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:43.018356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:43.020563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:43.044037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 823/1642] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:42 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:17:43.211181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd91544-ab52-4f78-be5f-d3e0657df4bb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:43.222859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd91544-ab52-4f78-be5f-d3e0657df4bb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:43.229737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb93bc60-ce07-4928-9a7b-d451ee6807a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:43.230491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93bc60-ce07-4928-9a7b-d451ee6807a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:43.230881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93bc60-ce07-4928-9a7b-d451ee6807a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:43.231223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93bc60-ce07-4928-9a7b-d451ee6807a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:43.249309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb93bc60-ce07-4928-9a7b-d451ee6807a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:43.251080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb93bc60-ce07-4928-9a7b-d451ee6807a0 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:43.281713 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 820/1643] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:43 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:43.452879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:43.668762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed7cd76-9c7d-44a7-8757-8eb42352fdba service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:43.688523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:43.690242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:43.690657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:43.691434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] PATH_INFO: `/v3/users/d7e1763c88334c29a14bb192bfe32bb2` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:43.695877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7e1763c88334c29a14bb192bfe32bb2)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:43.701715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:43.837744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7e1763c88334c29a14bb192bfe32bb2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:43.838560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7e1763c88334c29a14bb192bfe32bb2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:43.850499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7e1763c88334c29a14bb192bfe32bb2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:43.851267 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7e1763c88334c29a14bb192bfe32bb2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:43.867622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7e1763c88334c29a14bb192bfe32bb2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:43.917061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 824/1644] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:43 2022] DELETE /identity/v3/users/d7e1763c88334c29a14bb192bfe32bb2 => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:43.918469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:43.918931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:43.919102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:43.989997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:43.992532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:44.166596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 821/1645] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:44.222664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:44.227209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:44.269676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:44.272502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:44.304273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a41808-f8e9-4937-b8eb-55191206762a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:44.312812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:44.314949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:44.315112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:44.315285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] PATH_INFO: `/v3/users/791c81198cc74fac808d2b40209d7a32` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:44.318216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=791c81198cc74fac808d2b40209d7a32)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:44.320281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: WARNING py.warnings [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:17:44.320281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: warnings.warn(msg) Oct 01 04:17:44.320281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: Oct 01 04:17:44.324050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:44.388332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d348047-12b8-4530-bd70-defe5682ee84 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:44.397837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:44.400524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:44.402103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:44.448091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:44.451758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:44.452855 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:44.457671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:44.464575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '791c81198cc74fac808d2b40209d7a32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.490073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 825/1646] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:44.493746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '791c81198cc74fac808d2b40209d7a32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.495459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '791c81198cc74fac808d2b40209d7a32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.496311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '791c81198cc74fac808d2b40209d7a32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.513005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:44.516257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:44.520200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-eb511617-8a4a-4368-8236-2b2ff2eeecd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '791c81198cc74fac808d2b40209d7a32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.577075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4747afd-d618-4d13-8615-33dfc64e69f3 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:44.591009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:44.597101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:44.597188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:44.597317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 822/1647] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:44 2022] DELETE /identity/v3/users/791c81198cc74fac808d2b40209d7a32 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:44.597542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] PATH_INFO: `/v3/users/9e394c96cb604d0d8a4f53a8403d53ac` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:44.598918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d7ce0-580f-4fa3-939f-b4f026171c68 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:44.599266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d7ce0-580f-4fa3-939f-b4f026171c68 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:44.599551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401d7ce0-580f-4fa3-939f-b4f026171c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:44.600605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e394c96cb604d0d8a4f53a8403d53ac)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:44.604826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:44.621800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:44.626779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-401d7ce0-580f-4fa3-939f-b4f026171c68 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:44.628475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-401d7ce0-580f-4fa3-939f-b4f026171c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:44.711033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e394c96cb604d0d8a4f53a8403d53ac'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.711740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e394c96cb604d0d8a4f53a8403d53ac'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.726740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e394c96cb604d0d8a4f53a8403d53ac'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.727509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e394c96cb604d0d8a4f53a8403d53ac'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.740614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e394c96cb604d0d8a4f53a8403d53ac'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:44.751938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 823/1648] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:44.771403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 826/1649] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:44 2022] DELETE /identity/v3/users/9e394c96cb604d0d8a4f53a8403d53ac => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:17:44.773432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:44.773727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:44.774014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:44.796667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:44.798486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:44.824231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:44.827708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:44.952330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 824/1650] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:45.021832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753ccfd6-70b0-4594-ac89-138d9bf07a60 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:45.031243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:45.031883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:45.032164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:45.033196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:45.044663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:45.046531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:45.057225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:45.058676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:45.070148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7813b-2268-4557-8386-53bf3f135b05 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:45.080919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 827/1651] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:45.103143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:45.103871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:45.104150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:45.104420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:45.120428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:45.122154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:45.138843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 825/1652] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:46.483163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:46.485683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:46.517604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10379072-bdd7-458f-9cd5-5e46e305f20a service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:46.536006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:46.536654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:46.536654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:46.536654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:46.536654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] PATH_INFO: `/v3/projects/3ebfba30505f48c181cb04960d38aa63` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:46.537338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:46.538186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ebfba30505f48c181cb04960d38aa63)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:46.539034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: WARNING py.warnings [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 01 04:17:46.539034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: warnings.warn(msg) Oct 01 04:17:46.539034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: Oct 01 04:17:46.540242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:46.563388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaca061-f08c-4935-9d43-2f7628a8bb4d service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:46.578218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:46.580499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:46.580499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:46.580499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] PATH_INFO: `/v3/users/bbe1637bd0df445b87f04561ccbbaafe` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:46.582896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbe1637bd0df445b87f04561ccbbaafe)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:46.585166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:46.694449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:46.706174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbe1637bd0df445b87f04561ccbbaafe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.711427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ebfba30505f48c181cb04960d38aa63', 'request_id': 'req-260b3ae1-b245-4d0e-99c8-c13a82410c34'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.731116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ebfba30505f48c181cb04960d38aa63', 'request_id': 'req-260b3ae1-b245-4d0e-99c8-c13a82410c34'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.731769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbe1637bd0df445b87f04561ccbbaafe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.734938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbe1637bd0df445b87f04561ccbbaafe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.736608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbe1637bd0df445b87f04561ccbbaafe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.747495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ebfba30505f48c181cb04960d38aa63', 'request_id': 'req-260b3ae1-b245-4d0e-99c8-c13a82410c34'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.754373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a6f819f3-d0d2-4409-a331-e23a8da74250 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbe1637bd0df445b87f04561ccbbaafe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.795511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 826/1653] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:46 2022] DELETE /identity/v3/users/bbe1637bd0df445b87f04561ccbbaafe => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:17:46.796911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] The token cache is being invalidate because project 3ebfba30505f48c181cb04960d38aa63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:17:46.797349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.798778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-260b3ae1-b245-4d0e-99c8-c13a82410c34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:46.800567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:46.800978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:46.801349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:46.806579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 828/1654] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:17:46 2022] DELETE /identity/v3/projects/3ebfba30505f48c181cb04960d38aa63 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:46.811195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc9d08-dfbb-45fb-bc3a-1067cee5680b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:46.811566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc9d08-dfbb-45fb-bc3a-1067cee5680b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:46.811910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc9d08-dfbb-45fb-bc3a-1067cee5680b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:46.844877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:46.847926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:46.855780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-63fc9d08-dfbb-45fb-bc3a-1067cee5680b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:46.858860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-63fc9d08-dfbb-45fb-bc3a-1067cee5680b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:46.873207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:46.987321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 829/1655] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:46.995246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:46.995764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:47.007626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:47.009177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 827/1656] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:47.013245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:47.019510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.042204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a037ed38-24d7-4102-8cf1-88e2921d4570 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:47.052203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:47.054118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:47.054811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:47.055172 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:47.055518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] PATH_INFO: `/v3/projects/f4fe4437a4ee4d8f86d3834942b1fe3d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:47.057397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.058199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4fe4437a4ee4d8f86d3834942b1fe3d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:47.060244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:47.080017 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 830/1657] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:47.081074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:47.083840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.169283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4fe4437a4ee4d8f86d3834942b1fe3d', 'request_id': 'req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.183374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4fe4437a4ee4d8f86d3834942b1fe3d', 'request_id': 'req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.192888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4fe4437a4ee4d8f86d3834942b1fe3d', 'request_id': 'req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.215858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093ba852-19a1-4912-a63a-7fa66b82d792 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:47.219558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] The token cache is being invalidate because project f4fe4437a4ee4d8f86d3834942b1fe3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:17:47.222503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.223272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.229279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 828/1658] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:17:47 2022] DELETE /identity/v3/projects/f4fe4437a4ee4d8f86d3834942b1fe3d => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:47.229279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:47.233098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.335866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:47.339103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:47.339452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:47.340028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:47.342838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:47.345375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.374678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c00a5-7bb5-4918-bf94-b174c9b2823f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:47.385076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:47.386860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:47.388373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:47.388710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:47.389678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:47.389991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:47.396940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:47.433845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 829/1659] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:47.434898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:47.437057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.459506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e094305-03c9-4fd4-98b2-631c85c677e3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:47.476951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:47.477624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:47.477910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:47.478186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] PATH_INFO: `/v3/users/9690db63db3f4423bcdb7076008169ab` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:47.479417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9690db63db3f4423bcdb7076008169ab)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:47.481420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:47.505816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:47.507279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:47.540173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 831/1660] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:47.541298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:47.572967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51da1bb9-9335-4347-8fc8-27ebc9c94143 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:47.578399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:47.579024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:47.580007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:47.580286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:47.582084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:47.584054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:47.593998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9690db63db3f4423bcdb7076008169ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.604041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:47.621173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9690db63db3f4423bcdb7076008169ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.622540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9690db63db3f4423bcdb7076008169ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.624963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9690db63db3f4423bcdb7076008169ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.644896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 832/1661] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:47 2022] POST /identity/v3/projects => generated 373 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:47.648748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9690db63db3f4423bcdb7076008169ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:47.649119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:47.652714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.739115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 830/1662] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:47 2022] DELETE /identity/v3/users/9690db63db3f4423bcdb7076008169ab => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:47.790999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f172194f-a9a3-4f78-b363-86e5ceca172c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:47.884179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:47.885901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:47.925966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:47.926738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:47.927319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:47.927604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:47.928261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:47.930260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:47.976635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 833/1663] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:47 2022] POST /identity/v3/users => generated 287 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:17:47.980337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:48.007408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85858a6f-aba9-45c9-9b60-a70eb1cc1f2b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:48.016062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:48.018200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:48.018485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:48.020216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:48.021059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:48.025700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:48.033990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079560de-34de-466e-bfb8-98ef38ad6d43 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:48.042376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:48.043187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:48.043470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:48.043746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] PATH_INFO: `/v3/projects/1d594ba574fb4f8b8b97d4e3925b0d32` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:48.046743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d594ba574fb4f8b8b97d4e3925b0d32)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:48.055312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 834/1664] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:47 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:48.057479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:48.058828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:48.087250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2b1dae-e84b-4099-87b4-d68eb9896440 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:48.099294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:48.102306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:48.104171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:48.104471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:48.105229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:48.111426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:48.143652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 835/1665] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:48 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:48.148728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:48.177439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bc31c2-7a45-4127-9866-9dd32f8adba7 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:48.230715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d594ba574fb4f8b8b97d4e3925b0d32', 'request_id': 'req-1e12839d-8162-4152-815d-86aff1d0274c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.239065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:48.252626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d594ba574fb4f8b8b97d4e3925b0d32', 'request_id': 'req-1e12839d-8162-4152-815d-86aff1d0274c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.283129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d594ba574fb4f8b8b97d4e3925b0d32', 'request_id': 'req-1e12839d-8162-4152-815d-86aff1d0274c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.330133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] The token cache is being invalidate because project 1d594ba574fb4f8b8b97d4e3925b0d32 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:17:48.331053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.332320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.337739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 831/1666] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:17:47 2022] DELETE /identity/v3/projects/1d594ba574fb4f8b8b97d4e3925b0d32 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:48.341645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:48.346865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66adfe0-4615-4ef0-a38e-42263188b2f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:48.348274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:48.349007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66adfe0-4615-4ef0-a38e-42263188b2f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:48.349456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66adfe0-4615-4ef0-a38e-42263188b2f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:48.351265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66adfe0-4615-4ef0-a38e-42263188b2f7 admin admin] PATH_INFO: `/v3/projects/9a980fc83cca48e0ab131de04cc994b8/users/7473a37c4e3947ca956a2ba74b0ed229/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:48.381057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:48.408107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66adfe0-4615-4ef0-a38e-42263188b2f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a980fc83cca48e0ab131de04cc994b8, user_id=7473a37c4e3947ca956a2ba74b0ed229, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:48.409389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66adfe0-4615-4ef0-a38e-42263188b2f7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:48.425294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e12839d-8162-4152-815d-86aff1d0274c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:48.443525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 836/1667] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:48 2022] PUT /identity/v3/projects/9a980fc83cca48e0ab131de04cc994b8/users/7473a37c4e3947ca956a2ba74b0ed229/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:48.453251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:48.453567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:48.453816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:48.550156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] MFA Rules not processed for user `7473a37c4e3947ca956a2ba74b0ed229`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:48.551525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:48.647740 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:48.649097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:48.650035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:48.653597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] PATH_INFO: `/v3/users/92e78603d5d64f93847da5de8670ed5b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:48.656961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] RBAC: Authorizing `identity:delete_user(user_id=92e78603d5d64f93847da5de8670ed5b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:48.664247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:48.753394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 837/1668] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:48.757872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:48.759662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:48.787317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad70693d-9329-473d-9f6f-d11fc5dac198 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:48.814800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:48.815653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:48.815947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:48.816315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:48.817041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:48.819686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:48.863265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92e78603d5d64f93847da5de8670ed5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.864614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 838/1669] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:48 2022] POST /identity/v3/projects => generated 371 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:17:48.868337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:48.901557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92e78603d5d64f93847da5de8670ed5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.904289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92e78603d5d64f93847da5de8670ed5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.910261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92e78603d5d64f93847da5de8670ed5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.933037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92e78603d5d64f93847da5de8670ed5b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:48.995315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 832/1670] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:48 2022] DELETE /identity/v3/users/92e78603d5d64f93847da5de8670ed5b => generated 0 bytes in 654 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:49.036874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeb604d-2d05-4d74-9b97-68e0cc054b2d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:49.213656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:49.214464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:49.215006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:49.215304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:49.216242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:49.218377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:49.231754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:49.253074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 839/1671] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:48 2022] POST /identity/v3/users => generated 292 bytes in 385 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:17:49.256650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:49.258154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:49.285274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4744ca0-7a9a-4567-b114-6df7cfc8c315 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:49.306655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:49.307311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:49.307601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:49.307880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:49.308549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:49.310233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:49.331990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 833/1672] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:49 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:49.336313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:49.350230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a20e611-4a69-432b-aea6-28c9119e5186 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:49.367559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:49.367559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:49.367559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:49.367888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] PATH_INFO: `/v3/projects/065925f0023e436d8e045acd3617d12d/users/ab157d08158349c7b292955165982618/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:49.399597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] RBAC: Authorizing `identity:create_grant(project_id=065925f0023e436d8e045acd3617d12d, user_id=ab157d08158349c7b292955165982618, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:49.402151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:49.428245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 840/1673] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:49 2022] PUT /identity/v3/projects/065925f0023e436d8e045acd3617d12d/users/ab157d08158349c7b292955165982618/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:49.429342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:49.447696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c146a12e-c776-4ccb-bba3-b058c3d6c7f2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:49.566414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:49.597878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b27e193-fb8e-4e2f-bdc4-462a46671dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:49.598710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b27e193-fb8e-4e2f-bdc4-462a46671dfa admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:49.599128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b27e193-fb8e-4e2f-bdc4-462a46671dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:49.599540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b27e193-fb8e-4e2f-bdc4-462a46671dfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:49.600345 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b27e193-fb8e-4e2f-bdc4-462a46671dfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:49.605787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b27e193-fb8e-4e2f-bdc4-462a46671dfa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:49.626358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 834/1674] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:49 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:49.632026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:49.665489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58925d53-22e8-448a-b089-170f556c55db admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:49.678480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:49.679173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:49.679457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:49.679734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] PATH_INFO: `/v3/projects/065925f0023e436d8e045acd3617d12d/users/ab157d08158349c7b292955165982618/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:49.683385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] RBAC: Authorizing `identity:create_grant(project_id=065925f0023e436d8e045acd3617d12d, user_id=ab157d08158349c7b292955165982618, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:49.686109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:49.710953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 841/1675] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:49 2022] PUT /identity/v3/projects/065925f0023e436d8e045acd3617d12d/users/ab157d08158349c7b292955165982618/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:17:49.724243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:49.724753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:49.724951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:49.816813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] MFA Rules not processed for user `ab157d08158349c7b292955165982618`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:49.821374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:50.100672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 835/1676] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:49 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 377 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:50.127173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:50.129119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:50.149796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:50.154147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:50.299102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3fa287-1ed0-4a1f-b0b5-571421e1356d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:50.303980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:50.304756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:50.307411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:50.307687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:50.332245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:50.335560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:50.347546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 842/1677] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:50 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:50.351441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c11d219-3abd-46e7-8f8d-8e8fbc6158cd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:50.371630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:50.372307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:50.372605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:50.372879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] PATH_INFO: `/v3/projects/9b206eb3f4e54053850c74d85123bb17` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:50.378193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b206eb3f4e54053850c74d85123bb17)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:50.380367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:50.499774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:50.518988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b206eb3f4e54053850c74d85123bb17', 'request_id': 'req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.534114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7be6b32-df5a-4ad1-9846-77092f4389b6 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:50.552201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b206eb3f4e54053850c74d85123bb17', 'request_id': 'req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.562342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b206eb3f4e54053850c74d85123bb17', 'request_id': 'req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.603418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:50.626333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] The token cache is being invalidate because project 9b206eb3f4e54053850c74d85123bb17 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:17:50.627484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.630787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.635520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 836/1678] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:17:50 2022] DELETE /identity/v3/projects/9b206eb3f4e54053850c74d85123bb17 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:50.650509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:50.655507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:50.689056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fe119d-2b0e-4664-b745-833c04d7476c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:50.697030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe119d-2b0e-4664-b745-833c04d7476c service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:50.697457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe119d-2b0e-4664-b745-833c04d7476c service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:50.697741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe119d-2b0e-4664-b745-833c04d7476c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:50.700754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e8fe119d-2b0e-4664-b745-833c04d7476c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:50.834844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85ddd1-5a6b-4b8a-ba55-1be504146b0e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:50.847797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:50.848776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:50.849295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:50.849951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] PATH_INFO: `/v3/projects/d741cb6333b846708b3d55abd77b3934` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:50.851374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] RBAC: Authorizing `identity:delete_project(project_id=d741cb6333b846708b3d55abd77b3934)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:50.853586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:50.865071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fe119d-2b0e-4664-b745-833c04d7476c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:50.868705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fe119d-2b0e-4664-b745-833c04d7476c service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:50.899563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 843/1679] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:17:50.930066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7d29cb-1d4a-406b-825f-81515e735b2e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:50.933698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7d29cb-1d4a-406b-825f-81515e735b2e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:50.940148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7d29cb-1d4a-406b-825f-81515e735b2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:50.946816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd741cb6333b846708b3d55abd77b3934', 'request_id': 'req-54052aee-7804-4b8a-8920-2926b18b2cad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.963618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd741cb6333b846708b3d55abd77b3934', 'request_id': 'req-54052aee-7804-4b8a-8920-2926b18b2cad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.976245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd741cb6333b846708b3d55abd77b3934', 'request_id': 'req-54052aee-7804-4b8a-8920-2926b18b2cad'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:50.995400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-fa7d29cb-1d4a-406b-825f-81515e735b2e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:50.996640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fa7d29cb-1d4a-406b-825f-81515e735b2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.005355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:51.007121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] The token cache is being invalidate because project d741cb6333b846708b3d55abd77b3934 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:17:51.007451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:51.012456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-54052aee-7804-4b8a-8920-2926b18b2cad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:51.014681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 837/1680] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:17:50 2022] DELETE /identity/v3/projects/d741cb6333b846708b3d55abd77b3934 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:51.020400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.020698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.021296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.078925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:51.082459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.186263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 844/1681] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:51.187792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e5571a-b738-4153-bcb4-fce75cc80830 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.188077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e5571a-b738-4153-bcb4-fce75cc80830 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.188953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e5571a-b738-4153-bcb4-fce75cc80830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.245297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 838/1682] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:51.246521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:51.254540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-45e5571a-b738-4153-bcb4-fce75cc80830 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:51.257247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-45e5571a-b738-4153-bcb4-fce75cc80830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.258238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.283483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 845/1683] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:51.288090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0ea57-5953-4403-be54-ac156d2f9cd1 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.288401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0ea57-5953-4403-be54-ac156d2f9cd1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.288679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0ea57-5953-4403-be54-ac156d2f9cd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.289159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580eab6b-ad03-4fb7-9538-a81617ad3571 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:51.303096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:51.303866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.304150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.304424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] PATH_INFO: `/v3/projects/313646c210bd4deb80ce02228c46a29a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.305639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=313646c210bd4deb80ce02228c46a29a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:51.309239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:51.349586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-98f0ea57-5953-4403-be54-ac156d2f9cd1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:51.350820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-98f0ea57-5953-4403-be54-ac156d2f9cd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.367134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:51.380144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 846/1684] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:51.380144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0ea57-5953-4403-be54-ac156d2f9cd1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:51.405935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0ea57-5953-4403-be54-ac156d2f9cd1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:51.421831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '313646c210bd4deb80ce02228c46a29a', 'request_id': 'req-a10f2cb1-072c-4a74-9b29-bb339bb11f69'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:51.445596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '313646c210bd4deb80ce02228c46a29a', 'request_id': 'req-a10f2cb1-072c-4a74-9b29-bb339bb11f69'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:51.469246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '313646c210bd4deb80ce02228c46a29a', 'request_id': 'req-a10f2cb1-072c-4a74-9b29-bb339bb11f69'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:51.508753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] The token cache is being invalidate because project 313646c210bd4deb80ce02228c46a29a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:17:51.509046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:51.518463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:51.522696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 839/1685] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:17:51 2022] DELETE /identity/v3/projects/313646c210bd4deb80ce02228c46a29a => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:51.526656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:51.532171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.561878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f168c9ef-86c4-45ff-ac0d-884e89d3f726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:51.563104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168c9ef-86c4-45ff-ac0d-884e89d3f726 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.564361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168c9ef-86c4-45ff-ac0d-884e89d3f726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.564555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f168c9ef-86c4-45ff-ac0d-884e89d3f726 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.567469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f168c9ef-86c4-45ff-ac0d-884e89d3f726 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:51.567732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f168c9ef-86c4-45ff-ac0d-884e89d3f726 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:51.581008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f168c9ef-86c4-45ff-ac0d-884e89d3f726 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:51.614303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 847/1686] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:51.615225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.615515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.615960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.660880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:51.666803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.709782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 848/1687] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:51.720057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:51.731658 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.749429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f2cb1-072c-4a74-9b29-bb339bb11f69 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:51.767873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5fabf6-6aec-42e0-8a12-2d6ddb5d6582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:51.768552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5fabf6-6aec-42e0-8a12-2d6ddb5d6582 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.768838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5fabf6-6aec-42e0-8a12-2d6ddb5d6582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.771989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5fabf6-6aec-42e0-8a12-2d6ddb5d6582 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.773138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5fabf6-6aec-42e0-8a12-2d6ddb5d6582 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:51.773462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5fabf6-6aec-42e0-8a12-2d6ddb5d6582 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:51.774394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c949e2-097f-4fb6-9029-943bbff469bd None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:51.776627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5fabf6-6aec-42e0-8a12-2d6ddb5d6582 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:51.792096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:51.792741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.793202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.793481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.794194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:51.796279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:51.813221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 840/1688] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:51.815660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298b992e-00bf-4535-9101-28da530128dd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:51.815955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298b992e-00bf-4535-9101-28da530128dd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:51.816237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298b992e-00bf-4535-9101-28da530128dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:51.870780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 849/1689] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/projects => generated 363 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:51.871637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-298b992e-00bf-4535-9101-28da530128dd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:51.872854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-298b992e-00bf-4535-9101-28da530128dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:51.873705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:51.994573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 841/1690] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:51.994573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298b992e-00bf-4535-9101-28da530128dd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:51.999923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f5f0c-c9f0-4f75-aba1-8f22f9d6ad62 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.006098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.006591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298b992e-00bf-4535-9101-28da530128dd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.011648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.012376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.013131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.013461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.013901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.014105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.014524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.014761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.015486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.017180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.018390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.037136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:52.061360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 842/1691] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/users => generated 282 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:52.062288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.074265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa18269e-6627-4bd8-94c7-1db1fe6c7ee4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.079581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 850/1692] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:51 2022] POST /identity/v3/projects => generated 403 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:52.081169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.122188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:52.179042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eedcda0-9c10-46a8-94cc-48ebe0fa0365 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.185217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.187463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.188336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.189409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.190461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.195735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.201350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.202734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.205709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.205709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.208486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:17:52.208813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.213256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.214193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 851/1693] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:52 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:52.215319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.231821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec25b1-8b92-48bf-bf4a-246be91a8012 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.238714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.239285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.239953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.239953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.240529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.243044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.249624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 843/1694] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:17:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:17:52.250557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.273949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a8cf90-954d-4503-ad2e-c814e4e6ff99 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.280759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 852/1695] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:52 2022] POST /identity/v3/users => generated 302 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:52.280759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.281542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.282257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.282543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.282822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.283474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.286040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.297567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52138a5-93f4-4d13-bff1-947912b81d4c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.305483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 844/1696] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:52 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:52.306493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.316891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.317578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.317859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.318132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.318775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.319736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11cd55-f437-4187-8350-dd9ce9a98048 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.320681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.331251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.332209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.332622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.333083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.333873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.335598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.359263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 845/1697] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:52 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:52.360588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.376596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626c7e00-c40e-4319-8b3e-60f08544967d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.381376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.382058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.385444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.385726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] PATH_INFO: `/v3/projects/79facf999fdd48ada2b00a4d98a99df7/users/ec8132766d854bdf8d05972e221101cc/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.393000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 853/1698] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:52 2022] POST /identity/v3/projects => generated 353 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:52.394254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.413490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79facf999fdd48ada2b00a4d98a99df7, user_id=ec8132766d854bdf8d05972e221101cc, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.416383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.442900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 846/1699] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:52 2022] PUT /identity/v3/projects/79facf999fdd48ada2b00a4d98a99df7/users/ec8132766d854bdf8d05972e221101cc/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:52.444064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.544657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b53a1b-54ee-4e56-abf6-50b4b73108a6 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.548201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9f9dad-a12f-48c3-aa86-93809e3210e0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.554295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.555329 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.555745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.556240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.557000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bece1a8-ed41-4db6-9078-88342a679fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.557241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.557830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bece1a8-ed41-4db6-9078-88342a679fa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.558203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bece1a8-ed41-4db6-9078-88342a679fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.558580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bece1a8-ed41-4db6-9078-88342a679fa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.559333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bece1a8-ed41-4db6-9078-88342a679fa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.559602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.566172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bece1a8-ed41-4db6-9078-88342a679fa7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.596670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 854/1700] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:52 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:52.597543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17137d87-dadc-4334-92ba-3b1e8613b8bd None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.597848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17137d87-dadc-4334-92ba-3b1e8613b8bd None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.598301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 847/1701] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:52 2022] POST /identity/v3/users => generated 277 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:52.599162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.599787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17137d87-dadc-4334-92ba-3b1e8613b8bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.619746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb5c347-1e3c-453d-94d0-3aa1be020f14 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.634710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.635412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.635691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.635962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] PATH_INFO: `/v3/projects/cb3729754cf8451e869578e3a6817e5f/users/962037e1e781490cbaab1bec2feb18ee/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.669189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-17137d87-dadc-4334-92ba-3b1e8613b8bd None None] MFA Rules not processed for user `ec8132766d854bdf8d05972e221101cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:52.673732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-17137d87-dadc-4334-92ba-3b1e8613b8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:52.678692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb3729754cf8451e869578e3a6817e5f, user_id=962037e1e781490cbaab1bec2feb18ee, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.685988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.721189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 848/1702] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:52 2022] PUT /identity/v3/projects/cb3729754cf8451e869578e3a6817e5f/users/962037e1e781490cbaab1bec2feb18ee/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:52.722048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.740625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:52.827664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 855/1703] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:52 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:52.827664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241eb1ec-8bfa-4871-bf5c-bda0dcf17757 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.827664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241eb1ec-8bfa-4871-bf5c-bda0dcf17757 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.827664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241eb1ec-8bfa-4871-bf5c-bda0dcf17757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.829276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:52.865683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1105d3-2356-444d-a437-9667463ffd5b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.877700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.879985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.880419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.881482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.884613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.886549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:52.910446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-241eb1ec-8bfa-4871-bf5c-bda0dcf17757 None None] MFA Rules not processed for user `962037e1e781490cbaab1bec2feb18ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:52.911857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-241eb1ec-8bfa-4871-bf5c-bda0dcf17757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:52.917932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 849/1704] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:52 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:52.918776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:52.947885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24148929-ecd8-4cfe-8731-add79822ceba admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:52.961221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:52.962425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:52.962784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:52.963642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:52.964301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:52.969505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.027620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 850/1705] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:52 2022] POST /identity/v3/projects => generated 365 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:53.028594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.055076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 856/1706] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:52 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:53.056242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241eb1ec-8bfa-4871-bf5c-bda0dcf17757 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.085258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241eb1ec-8bfa-4871-bf5c-bda0dcf17757 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.177039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15e636d-d618-4721-8a91-51617a83c5fc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.183272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.183912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.184250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.184534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.185965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.187785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.207799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 851/1707] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:53 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:53.209316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.217724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.218685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.219159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.220328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.222603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c52629-6ffe-4481-bfb0-9f104b5338e3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.224241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.226448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.228613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.229454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.229755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.230034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.231005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.234472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.280149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 857/1708] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:53 2022] POST /identity/v3/users => generated 289 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:53.281677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.291727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 852/1709] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:53 2022] POST /identity/v3/projects => generated 403 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:53.293206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.304701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341f8a71-f58a-4c60-ab28-05f37e7fb6ae admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.410073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c651bba8-e178-4bee-8b2f-fc6ee2d36656 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.418941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.420259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.423636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.423766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.424468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:53.426480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.433927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.442434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.443274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.443652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.444021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] PATH_INFO: `/v3/projects/5f0e67b69acc4cb5844b183fd3176880/users/28c3860615d54b5680997c6634bc920f/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.459950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 853/1710] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:53 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:53.463482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f0e67b69acc4cb5844b183fd3176880, user_id=28c3860615d54b5680997c6634bc920f, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.464160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.465277 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.499529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 858/1711] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:53 2022] PUT /identity/v3/projects/5f0e67b69acc4cb5844b183fd3176880/users/28c3860615d54b5680997c6634bc920f/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:53.509415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.511017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef728506-ce4b-4449-b6ed-255bc0215b89 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.569377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:53.684854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b0f53-df6b-4b1c-9fc0-77f4f951e780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.685502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b0f53-df6b-4b1c-9fc0-77f4f951e780 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.685778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b0f53-df6b-4b1c-9fc0-77f4f951e780 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.686053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b0f53-df6b-4b1c-9fc0-77f4f951e780 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.688821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077b0f53-df6b-4b1c-9fc0-77f4f951e780 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.691027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077b0f53-df6b-4b1c-9fc0-77f4f951e780 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.697814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973f5d4e-81da-4b8f-9847-52d6e9719c2d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.709705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.711374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.711374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.711374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] PATH_INFO: `/v3/projects/ffbe465f13f04c19b58b10aeb02886a8/users/8d97bed740b84a34acc7f21127452cc5/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.751648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 854/1712] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:53 2022] POST /identity/v3/users => generated 308 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:53.751648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23a9f0b-5275-4112-a563-3eceb4c30c96 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.751648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23a9f0b-5275-4112-a563-3eceb4c30c96 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.751648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23a9f0b-5275-4112-a563-3eceb4c30c96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.760885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffbe465f13f04c19b58b10aeb02886a8, user_id=8d97bed740b84a34acc7f21127452cc5, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.764691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.814814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 859/1713] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:53 2022] PUT /identity/v3/projects/ffbe465f13f04c19b58b10aeb02886a8/users/8d97bed740b84a34acc7f21127452cc5/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:53.814814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:53.841785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b23a9f0b-5275-4112-a563-3eceb4c30c96 None None] MFA Rules not processed for user `28c3860615d54b5680997c6634bc920f`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:53.843005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b23a9f0b-5275-4112-a563-3eceb4c30c96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:53.952108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f82c900-4cbb-4ded-ba2d-fc7e5d94d83d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:53.963930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:53.966109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:53.966380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:53.966652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:53.970080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:53.971704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:53.984395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 855/1714] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:53 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:53.985935 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23a9f0b-5275-4112-a563-3eceb4c30c96 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.010755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 860/1715] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:53 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:54.013971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.035488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23a9f0b-5275-4112-a563-3eceb4c30c96 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.044514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d08e4e-ec0b-4fd5-a07e-e9d58b56caba admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.054835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.055875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.056522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.056789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.057086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.058859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.059373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.059842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.060320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.060917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.061791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.070043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.089650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 856/1716] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:53 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:54.090686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.113975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd85aaba-82d6-4cd2-b3d7-c6a445e02543 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.124257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 861/1717] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:17:54 2022] POST /identity/v3/projects => generated 355 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:54.125388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.130397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.131084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.131366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.131641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] PATH_INFO: `/v3/projects/447d62793dce411dadc1dc57b6a12fff/users/f65e508434e742179dacfddbdd1bcf79/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.171676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=447d62793dce411dadc1dc57b6a12fff, user_id=f65e508434e742179dacfddbdd1bcf79, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.174777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.192589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:54.212047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 857/1718] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:54 2022] PUT /identity/v3/projects/447d62793dce411dadc1dc57b6a12fff/users/f65e508434e742179dacfddbdd1bcf79/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:54.214196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.256962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f9570e-0e63-4fd2-ac69-50e5b722db65 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.310958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb5888-b3c7-4dee-a407-eb7e04d21e70 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.318147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.319107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.319598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.327311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.329576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.333155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.363045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:54.373790 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 858/1719] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:17:54 2022] POST /identity/v3/users => generated 284 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:17:54.376814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.385207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.397883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883c1e85-f993-48c3-8a4f-3a6a30063ef3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.398324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.398774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.399187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] PATH_INFO: `/v3/projects/ffbe465f13f04c19b58b10aeb02886a8/users/8d97bed740b84a34acc7f21127452cc5/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.403030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffbe465f13f04c19b58b10aeb02886a8, user_id=8d97bed740b84a34acc7f21127452cc5, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.408979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.412212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2064b9b-8ba5-4edf-9749-74d9a597daf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.412962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2064b9b-8ba5-4edf-9749-74d9a597daf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.414046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2064b9b-8ba5-4edf-9749-74d9a597daf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.414767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2064b9b-8ba5-4edf-9749-74d9a597daf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.417440 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2064b9b-8ba5-4edf-9749-74d9a597daf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.422049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2064b9b-8ba5-4edf-9749-74d9a597daf8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.436265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 862/1720] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:54 2022] PUT /identity/v3/projects/ffbe465f13f04c19b58b10aeb02886a8/users/8d97bed740b84a34acc7f21127452cc5/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:54.437731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.452058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 859/1721] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:54 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:54.456211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671ec6be-3bfc-4448-9bec-fecd4225d0ee None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.457788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671ec6be-3bfc-4448-9bec-fecd4225d0ee None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.458753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671ec6be-3bfc-4448-9bec-fecd4225d0ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.528697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-671ec6be-3bfc-4448-9bec-fecd4225d0ee None None] MFA Rules not processed for user `8d97bed740b84a34acc7f21127452cc5`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:54.530053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-671ec6be-3bfc-4448-9bec-fecd4225d0ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:54.544123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae9fa42-f50d-4a54-8c95-6fc5f96f1225 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.556940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.557512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.557790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.558060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.558868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.560333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.587109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 863/1722] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:54 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:54.588050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.604974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7450eb91-1594-4358-bd60-19179f13b4fe admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.620606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.621382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.621745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.622026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] PATH_INFO: `/v3/projects/447d62793dce411dadc1dc57b6a12fff/users/f65e508434e742179dacfddbdd1bcf79/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.625052 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] RBAC: Authorizing `identity:create_grant(project_id=447d62793dce411dadc1dc57b6a12fff, user_id=f65e508434e742179dacfddbdd1bcf79, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.630405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.662239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 864/1723] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:54 2022] PUT /identity/v3/projects/447d62793dce411dadc1dc57b6a12fff/users/f65e508434e742179dacfddbdd1bcf79/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:54.663736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.690270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 860/1724] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:54 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:54.691454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad072d-b62c-4be7-ad21-bb1941d75e5d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.691744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad072d-b62c-4be7-ad21-bb1941d75e5d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.692021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad072d-b62c-4be7-ad21-bb1941d75e5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.764030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a6ad072d-b62c-4be7-ad21-bb1941d75e5d None None] MFA Rules not processed for user `f65e508434e742179dacfddbdd1bcf79`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:54.765472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a6ad072d-b62c-4be7-ad21-bb1941d75e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:54.829493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:54.842886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad32502-3104-45e4-8569-2ac4a5f90184 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:54.856215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:54.856907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:54.857207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:54.857479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] PATH_INFO: `/v3/projects/d1f02d5f57204f339652485e863eddd4/users/6b5dc5392a46420cbe4586138c3fb573/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:54.909852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1f02d5f57204f339652485e863eddd4, user_id=6b5dc5392a46420cbe4586138c3fb573, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:54.912749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:54.940030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 865/1725] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:54 2022] PUT /identity/v3/projects/d1f02d5f57204f339652485e863eddd4/users/6b5dc5392a46420cbe4586138c3fb573/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:54.942525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:54.948985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:54.954838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 861/1726] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:54 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:54.957588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ad072d-b62c-4be7-ad21-bb1941d75e5d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:55.003570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ad072d-b62c-4be7-ad21-bb1941d75e5d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:55.099541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bafefd0-38fc-40b9-adc3-c6f86a38f408 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:55.114149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105000b6-ffd3-4a22-b5f5-c908c2a42694 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:55.115114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105000b6-ffd3-4a22-b5f5-c908c2a42694 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.115390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105000b6-ffd3-4a22-b5f5-c908c2a42694 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.115663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105000b6-ffd3-4a22-b5f5-c908c2a42694 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.141688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:55.142574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.142863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.143142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.143855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:55.145662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:55.147777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:55.169044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 862/1727] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:17:54 2022] GET /identity/v3/roles => generated 1420 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:17:55.170656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:55.176827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:55.205657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45102aa3-731b-47b5-abd1-f7f0506f1ed9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:55.218112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:55.218744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.219025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.219297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.267720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105000b6-ffd3-4a22-b5f5-c908c2a42694 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:55.269755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105000b6-ffd3-4a22-b5f5-c908c2a42694 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:55.296728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 866/1728] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:54 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:55.301568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40198d5f-2fea-4ca5-9ce1-aef420a71001 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.303729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40198d5f-2fea-4ca5-9ce1-aef420a71001 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.303729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40198d5f-2fea-4ca5-9ce1-aef420a71001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.333621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:55.335432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:55.370052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 863/1729] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:55 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:55.372971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:55.377266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-40198d5f-2fea-4ca5-9ce1-aef420a71001 None None] MFA Rules not processed for user `bfcc0e7f557e4334a808f5dfaffe93db`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:55.378405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-40198d5f-2fea-4ca5-9ce1-aef420a71001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:55.397065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc477a-807f-41ba-ba75-cb7607062616 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:55.404267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2b5ded-581c-473c-8e63-bf9cd00fd797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:55.405125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b5ded-581c-473c-8e63-bf9cd00fd797 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.405597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b5ded-581c-473c-8e63-bf9cd00fd797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.405971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b5ded-581c-473c-8e63-bf9cd00fd797 admin admin] PATH_INFO: `/v3/projects/d1f02d5f57204f339652485e863eddd4/users/6b5dc5392a46420cbe4586138c3fb573/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.418619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2b5ded-581c-473c-8e63-bf9cd00fd797 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1f02d5f57204f339652485e863eddd4, user_id=6b5dc5392a46420cbe4586138c3fb573, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:55.421666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2b5ded-581c-473c-8e63-bf9cd00fd797 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:55.464398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 864/1730] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:17:55 2022] PUT /identity/v3/projects/d1f02d5f57204f339652485e863eddd4/users/6b5dc5392a46420cbe4586138c3fb573/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:55.467320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16419fb1-5ae8-4ed5-984f-825d33400262 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.467900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16419fb1-5ae8-4ed5-984f-825d33400262 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.468565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16419fb1-5ae8-4ed5-984f-825d33400262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.570284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-16419fb1-5ae8-4ed5-984f-825d33400262 None None] MFA Rules not processed for user `6b5dc5392a46420cbe4586138c3fb573`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:55.572262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-16419fb1-5ae8-4ed5-984f-825d33400262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:55.605486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 867/1731] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:55 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:55.606714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40198d5f-2fea-4ca5-9ce1-aef420a71001 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:55.640264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40198d5f-2fea-4ca5-9ce1-aef420a71001 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:55.678721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:55.707517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 865/1732] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:55 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:55.712806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.713558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.713558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.746083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] MFA Rules not processed for user `bfcc0e7f557e4334a808f5dfaffe93db`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:55.753200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:55.762809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:55.764881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:55.765105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:55.765432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:55.882307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 866/1733] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:55 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:55.892284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:55.894357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:55.908913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:55.912081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:55.917570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:55.949439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 868/1734] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:55 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:56.113787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe13979-8019-40c0-a902-ff3eb6630c3f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:56.134447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:56.135725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:56.135725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:56.136007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:56.146229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:56.148139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:56.157995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:56.168167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 867/1735] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:55 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:17:56.179098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:56.214842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39a7b9-0f3c-4491-84aa-20eb6de6d1bc service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:56.223262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:56.224008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:56.225978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:56.226271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:56.253268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:56.257113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:56.278409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 868/1736] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:56 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:17:56.299720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30aa2323-53ad-44f5-a68a-ed5d5252de74 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:56.310088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:56.311210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:56.311418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:56.312674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] PATH_INFO: `/v3/users/962037e1e781490cbaab1bec2feb18ee` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:56.315326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] RBAC: Authorizing `identity:delete_user(user_id=962037e1e781490cbaab1bec2feb18ee)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:56.335370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:56.428467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '962037e1e781490cbaab1bec2feb18ee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:56.428856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:56.429952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '962037e1e781490cbaab1bec2feb18ee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:56.437730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a5b9a5-bdc2-4e65-bcb8-180731f5c881 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:56.441065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '962037e1e781490cbaab1bec2feb18ee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:56.441905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '962037e1e781490cbaab1bec2feb18ee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:56.452355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85ec6685-02d1-45af-919c-cc6b1f2e5501 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '962037e1e781490cbaab1bec2feb18ee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:56.455334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:56.456475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:56.456475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:56.456475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:56.458572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:56.466028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:56.492653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 869/1737] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:56 2022] DELETE /identity/v3/users/962037e1e781490cbaab1bec2feb18ee => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:56.496752 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9388da7-d74c-464d-95b5-de159414566d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:56.497331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9388da7-d74c-464d-95b5-de159414566d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:56.497859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9388da7-d74c-464d-95b5-de159414566d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:56.562738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a9388da7-d74c-464d-95b5-de159414566d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:56.567582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a9388da7-d74c-464d-95b5-de159414566d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:56.621265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:56.623696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:56.666443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 869/1738] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:56 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:56.738329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 870/1739] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:17:57.017570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:57.017570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:57.175898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a52335-7e05-49d1-9acf-86f65e028a88 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:57.273724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:57.341169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927915c-72d2-4942-a658-64a14c5c9b02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:57.341814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927915c-72d2-4942-a658-64a14c5c9b02 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:57.342096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927915c-72d2-4942-a658-64a14c5c9b02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:57.342371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927915c-72d2-4942-a658-64a14c5c9b02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:57.373074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d927915c-72d2-4942-a658-64a14c5c9b02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:57.374551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d927915c-72d2-4942-a658-64a14c5c9b02 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:57.388992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 870/1740] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:57.940158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23d5c6-867e-462d-9c70-ef2b6f0a3e7c None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:57.940646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23d5c6-867e-462d-9c70-ef2b6f0a3e7c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:57.941431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23d5c6-867e-462d-9c70-ef2b6f0a3e7c None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:57.943342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 871/1741] 10.222.0.26 () {54 vars in 912 bytes} [Sat Oct 1 04:17:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:17:57.948762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:57.949292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:57.950483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:58.052910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] MFA Rules not processed for user `16aab142c0a7445483caf19178d56062`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:58.056017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:58.081188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 871/1742] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:17:57 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:58.190520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d23d5c6-867e-462d-9c70-ef2b6f0a3e7c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:58.193042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7d23d5c6-867e-462d-9c70-ef2b6f0a3e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:58.274495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d23d5c6-867e-462d-9c70-ef2b6f0a3e7c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:58.283639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746c7fff-c531-4722-8ed6-6a250e153fc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:58.285302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c7fff-c531-4722-8ed6-6a250e153fc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:58.285783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c7fff-c531-4722-8ed6-6a250e153fc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:58.286241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c7fff-c531-4722-8ed6-6a250e153fc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:58.303054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746c7fff-c531-4722-8ed6-6a250e153fc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:58.306314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746c7fff-c531-4722-8ed6-6a250e153fc0 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:58.334965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 872/1743] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:17:58.769449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:58.772719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:58.820028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eae12d-4035-4258-99d4-1817e59161ff None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:58.836406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:58.840151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:58.840665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:58.841199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] PATH_INFO: `/v3/projects/cb3729754cf8451e869578e3a6817e5f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:58.845876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb3729754cf8451e869578e3a6817e5f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:58.850031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:58.968229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb3729754cf8451e869578e3a6817e5f', 'request_id': 'req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:58.980348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb3729754cf8451e869578e3a6817e5f', 'request_id': 'req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:58.994848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb3729754cf8451e869578e3a6817e5f', 'request_id': 'req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:58.998002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:58.998431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:58.998597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:59.036741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] The token cache is being invalidate because project cb3729754cf8451e869578e3a6817e5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:17:59.040150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:59.041948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:59.054082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 872/1744] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:17:58 2022] DELETE /identity/v3/projects/cb3729754cf8451e869578e3a6817e5f => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:17:59.058372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:59.065938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:59.076582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] MFA Rules not processed for user `3caf1cc366434d418c474d7a1879faa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:17:59.086869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:59.272363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb379fa-57fc-4f57-9e3d-6cb4b34f1159 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:59.281253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:59.282809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:59.283368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:59.283992 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] PATH_INFO: `/v3/users/f65e508434e742179dacfddbdd1bcf79` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:59.285917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f65e508434e742179dacfddbdd1bcf79)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:17:59.288623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:17:59.338115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 873/1745] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:17:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 341 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:17:59.376287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:59.377953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:59.414473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:59.438958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f65e508434e742179dacfddbdd1bcf79'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:59.484659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f65e508434e742179dacfddbdd1bcf79'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:59.487272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f65e508434e742179dacfddbdd1bcf79'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:59.490417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f65e508434e742179dacfddbdd1bcf79'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:59.521467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f65e508434e742179dacfddbdd1bcf79'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:17:59.592232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 873/1746] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:17:59 2022] DELETE /identity/v3/users/f65e508434e742179dacfddbdd1bcf79 => generated 0 bytes in 534 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:17:59.664647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42349b89-569a-4ef3-93e0-8bcd50258dfd None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:17:59.672103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:17:59.891765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:17:59.892405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:17:59.892819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:17:59.893081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:17:59.902426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:17:59.935059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:17:59.936648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:00.080663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:00.084422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:00.117944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 874/1747] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:59 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 742 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:00.120759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:00.129987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:00.196604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cf9ef3-13da-429d-a568-be8794034f69 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:00.212298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf7173e-bcc6-4488-b69a-624a7faae77a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:00.214095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf7173e-bcc6-4488-b69a-624a7faae77a service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:00.214380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf7173e-bcc6-4488-b69a-624a7faae77a service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:00.215786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf7173e-bcc6-4488-b69a-624a7faae77a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:00.222465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-daf7173e-bcc6-4488-b69a-624a7faae77a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:00.273294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf7173e-bcc6-4488-b69a-624a7faae77a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:00.283665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf7173e-bcc6-4488-b69a-624a7faae77a service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:00.318956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 874/1748] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:17:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:00.443174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a509778a-39a6-4073-86a0-5c426482d875 service nova] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:00.466933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:00.473019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:00.473370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:00.473806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] PATH_INFO: `/v3/users/ec8132766d854bdf8d05972e221101cc` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:00.476126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec8132766d854bdf8d05972e221101cc)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:00.491685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:00.643620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:00.647496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec8132766d854bdf8d05972e221101cc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:00.649547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec8132766d854bdf8d05972e221101cc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:00.678012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec8132766d854bdf8d05972e221101cc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:00.680530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec8132766d854bdf8d05972e221101cc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:00.703545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-25917e41-43e3-4d65-84fe-e588700c99c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec8132766d854bdf8d05972e221101cc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:00.847639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:00.848856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:00.849413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:00.850580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 875/1749] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:00 2022] DELETE /identity/v3/users/ec8132766d854bdf8d05972e221101cc => generated 0 bytes in 723 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:00.918087 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:00.923085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:01.148025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 875/1750] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:01.230098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:01.231058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:01.231687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:01.240237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:01.241720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:01.326985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] MFA Rules not processed for user `ccd631925a0a4f5ca8be5d3915778487`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:01.331907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:01.478920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:01.531309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f726ea1-0229-42c7-8799-4b30f15557bb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:01.559794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:01.563715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:01.564057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:01.564344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:01.606573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:01.616476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:01.654942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 876/1751] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:01.661810 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 876/1752] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:01 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 433 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:01.677848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:01.680317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:01.943264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019d4a21-739d-4ff9-9ede-1ea36bb78b8f service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:01.957748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0a6383-4b92-4e67-8cce-706e63bab896 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:01.959554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a6383-4b92-4e67-8cce-706e63bab896 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:01.959843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a6383-4b92-4e67-8cce-706e63bab896 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:01.960125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a6383-4b92-4e67-8cce-706e63bab896 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:01.982942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0a6383-4b92-4e67-8cce-706e63bab896 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:01.988674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0a6383-4b92-4e67-8cce-706e63bab896 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:02.018519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 877/1753] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:01 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:02.211435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:02.213490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:02.240140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d01ad6-5959-489b-9005-e4b35a8a3fbc None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:02.246976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:02.247851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:02.248336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:02.248749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] PATH_INFO: `/v3/projects/447d62793dce411dadc1dc57b6a12fff` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:02.250188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] RBAC: Authorizing `identity:delete_project(project_id=447d62793dce411dadc1dc57b6a12fff)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:02.252908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:02.417048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '447d62793dce411dadc1dc57b6a12fff', 'request_id': 'req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:02.442027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '447d62793dce411dadc1dc57b6a12fff', 'request_id': 'req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:02.465162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '447d62793dce411dadc1dc57b6a12fff', 'request_id': 'req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:02.502562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] The token cache is being invalidate because project 447d62793dce411dadc1dc57b6a12fff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:02.504859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:02.509565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d9f5374a-88a4-4f41-8e7b-2c78cbf9e912 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:02.514600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 877/1754] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:02 2022] DELETE /identity/v3/projects/447d62793dce411dadc1dc57b6a12fff => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:02.520819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9984b3c4-a81f-4b26-86d0-986733d9d80d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:02.521604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9984b3c4-a81f-4b26-86d0-986733d9d80d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:02.521604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9984b3c4-a81f-4b26-86d0-986733d9d80d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:02.562222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9984b3c4-a81f-4b26-86d0-986733d9d80d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:02.565452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9984b3c4-a81f-4b26-86d0-986733d9d80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:02.783608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a89c90-ddf4-42ab-be88-1cfa0fa8f61b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:02.784229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a89c90-ddf4-42ab-be88-1cfa0fa8f61b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:02.784734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a89c90-ddf4-42ab-be88-1cfa0fa8f61b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:02.790326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 878/1755] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:02.845154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e9a89c90-ddf4-42ab-be88-1cfa0fa8f61b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:02.846484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e9a89c90-ddf4-42ab-be88-1cfa0fa8f61b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:02.881824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9984b3c4-a81f-4b26-86d0-986733d9d80d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:02.891929 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 878/1756] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:02.900072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9984b3c4-a81f-4b26-86d0-986733d9d80d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:02.921037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:02.921686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:02.922885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:02.923199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:02.925348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:02.927355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:02.929119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:02.976519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a89c90-ddf4-42ab-be88-1cfa0fa8f61b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:02.977219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 879/1757] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:02.986943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a89c90-ddf4-42ab-be88-1cfa0fa8f61b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:02.996942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:02.997579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:02.997857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:02.998129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:02.998771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:03.001398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:03.077436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 879/1758] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:02 2022] POST /identity/v3/projects => generated 409 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:03.078218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:03.113688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a19555-7f06-415c-bcd8-9805281d9a31 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:03.228572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:03.229832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:03.230430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:03.230917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:03.232953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:03.236883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:03.250981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:03.302011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 880/1759] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:03 2022] POST /identity/v3/users => generated 305 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:03.306766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:03.327093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c0bcf-6a2d-4ab3-ba8f-aabe5a81ac10 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:03.349709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:03.350615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:03.351062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:03.351530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:03.352330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:03.355874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:03.364859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:03.404731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 880/1760] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:03 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:03.409357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:03.428226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b40592-09bc-4f1d-997a-212de68517cb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:03.446342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3632098c-c52f-4829-af65-d8675da5fe29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:03.455598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3632098c-c52f-4829-af65-d8675da5fe29 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:03.455947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3632098c-c52f-4829-af65-d8675da5fe29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:03.457396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3632098c-c52f-4829-af65-d8675da5fe29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:03.458133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3632098c-c52f-4829-af65-d8675da5fe29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:03.473181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3632098c-c52f-4829-af65-d8675da5fe29 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:03.541666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 881/1761] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:03 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:03.547028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:03.562852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9b982c-1bf8-4b0d-a81c-13615ca2ecaa admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:03.590257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7ea9de-34a5-44ea-b73f-0234babad969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:03.594350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7ea9de-34a5-44ea-b73f-0234babad969 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:03.595567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7ea9de-34a5-44ea-b73f-0234babad969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:03.598998 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7ea9de-34a5-44ea-b73f-0234babad969 admin admin] PATH_INFO: `/v3/projects/e75365d3c3d941fb8948ce24ba7dd2a9/users/e7245941dcba4ad7adcc9818fcf2a790/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:03.643592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7ea9de-34a5-44ea-b73f-0234babad969 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e75365d3c3d941fb8948ce24ba7dd2a9, user_id=e7245941dcba4ad7adcc9818fcf2a790, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:03.647335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7ea9de-34a5-44ea-b73f-0234babad969 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:03.708683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 881/1762] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:03 2022] PUT /identity/v3/projects/e75365d3c3d941fb8948ce24ba7dd2a9/users/e7245941dcba4ad7adcc9818fcf2a790/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:03.712246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a09a8a0-4a1f-451b-b0ab-cac7a94661fa None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:03.712536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a09a8a0-4a1f-451b-b0ab-cac7a94661fa None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:03.713066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a09a8a0-4a1f-451b-b0ab-cac7a94661fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:03.812593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71099620-d428-4c73-9baf-7a9dd5e33e2f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:03.814366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5a09a8a0-4a1f-451b-b0ab-cac7a94661fa None None] MFA Rules not processed for user `e7245941dcba4ad7adcc9818fcf2a790`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:03.815346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71099620-d428-4c73-9baf-7a9dd5e33e2f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:03.816677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71099620-d428-4c73-9baf-7a9dd5e33e2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:03.825107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5a09a8a0-4a1f-451b-b0ab-cac7a94661fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:04.019696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 882/1763] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:03 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:04.024981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a09a8a0-4a1f-451b-b0ab-cac7a94661fa None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:04.086725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-71099620-d428-4c73-9baf-7a9dd5e33e2f None None] MFA Rules not processed for user `7014a1defd81461787318bc27d56b9aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:04.096815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-71099620-d428-4c73-9baf-7a9dd5e33e2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:04.188363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a09a8a0-4a1f-451b-b0ab-cac7a94661fa None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:04.204669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:04.205328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:04.205627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:04.205901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:04.206546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:04.213301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:04.386340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 883/1764] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:04 2022] POST /identity/v3/projects => generated 409 bytes in 360 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:04.396893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:04.409574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:04.410543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 882/1765] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:03 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 601 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:04.412240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71099620-d428-4c73-9baf-7a9dd5e33e2f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:04.474105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71099620-d428-4c73-9baf-7a9dd5e33e2f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:04.662695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:04.672499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805bcc5b-80f0-4746-90d9-f355d549550e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:04.688521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:04.693586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:04.696354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:04.696354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] PATH_INFO: `/v3/projects/79facf999fdd48ada2b00a4d98a99df7` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:04.696354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79facf999fdd48ada2b00a4d98a99df7)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:04.698647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:04.749648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57393747-64d9-4697-9f3e-5a3ceb8a30ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:04.750829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57393747-64d9-4697-9f3e-5a3ceb8a30ba admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:04.751284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57393747-64d9-4697-9f3e-5a3ceb8a30ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:04.754055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57393747-64d9-4697-9f3e-5a3ceb8a30ba admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:04.759620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57393747-64d9-4697-9f3e-5a3ceb8a30ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:04.765434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57393747-64d9-4697-9f3e-5a3ceb8a30ba admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:04.856159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 883/1766] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:04 2022] POST /identity/v3/users => generated 311 bytes in 444 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:04.859351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ba19c-f3a3-4204-a74c-5996596f123a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:04.860217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ba19c-f3a3-4204-a74c-5996596f123a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:04.861011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ba19c-f3a3-4204-a74c-5996596f123a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:04.865011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79facf999fdd48ada2b00a4d98a99df7', 'request_id': 'req-efb997fd-373a-4fc9-9f00-931226ae0be4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:04.890168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79facf999fdd48ada2b00a4d98a99df7', 'request_id': 'req-efb997fd-373a-4fc9-9f00-931226ae0be4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:04.919537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79facf999fdd48ada2b00a4d98a99df7', 'request_id': 'req-efb997fd-373a-4fc9-9f00-931226ae0be4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:04.942278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-dd4ba19c-f3a3-4204-a74c-5996596f123a None None] MFA Rules not processed for user `7014a1defd81461787318bc27d56b9aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:04.944107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-dd4ba19c-f3a3-4204-a74c-5996596f123a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:04.961024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] The token cache is being invalidate because project 79facf999fdd48ada2b00a4d98a99df7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:04.961431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:04.962357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:04.967743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 884/1767] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:04 2022] DELETE /identity/v3/projects/79facf999fdd48ada2b00a4d98a99df7 => generated 0 bytes in 574 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:04.969799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:04.972788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:05.123176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb997fd-373a-4fc9-9f00-931226ae0be4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:05.129637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:05.130758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:05.131242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:05.132510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:05.134082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:05.136016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:05.158346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:05.169560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 885/1768] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:04 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:05.173337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:05.182472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:05.195110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 884/1769] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:04 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:05.199414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26b55d4-d42d-40d7-b715-60d5e2422bde None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:05.199414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26b55d4-d42d-40d7-b715-60d5e2422bde None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:05.199414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26b55d4-d42d-40d7-b715-60d5e2422bde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:05.227972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56756fe2-b8d0-4496-bb4d-0f90213f9e83 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:05.284540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:05.285242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:05.291566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:05.291905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] PATH_INFO: `/v3/users/8d97bed740b84a34acc7f21127452cc5` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:05.297858 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d97bed740b84a34acc7f21127452cc5)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:05.309556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:05.327329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d26b55d4-d42d-40d7-b715-60d5e2422bde None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:05.331852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d26b55d4-d42d-40d7-b715-60d5e2422bde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:05.499513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 885/1770] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:18:05 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:05.502183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26b55d4-d42d-40d7-b715-60d5e2422bde None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:05.513219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d97bed740b84a34acc7f21127452cc5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:05.542550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d97bed740b84a34acc7f21127452cc5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:05.548342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d97bed740b84a34acc7f21127452cc5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:05.551118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d97bed740b84a34acc7f21127452cc5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:05.555514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26b55d4-d42d-40d7-b715-60d5e2422bde None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:05.573834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d97bed740b84a34acc7f21127452cc5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:05.599068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:05.599769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:05.600047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:05.600333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] PATH_INFO: `/v3/projects/0e317b3e30d244afabeccc23c7933dd3/users/c8310a4d63774b5aad11a43165320c97/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:05.604574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:05.642326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 886/1771] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:05 2022] DELETE /identity/v3/users/8d97bed740b84a34acc7f21127452cc5 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:05.646472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:05.649902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:05.660646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e317b3e30d244afabeccc23c7933dd3, user_id=c8310a4d63774b5aad11a43165320c97, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:05.663364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:05.705372 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 886/1772] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:05 2022] PUT /identity/v3/projects/0e317b3e30d244afabeccc23c7933dd3/users/c8310a4d63774b5aad11a43165320c97/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:05.706600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:05.708546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:05.918424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33519bf6-1fa5-43ea-bbb2-e29b743df776 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:05.943272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fd6233-d24d-4003-8983-ca6b47a176e3 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:05.959984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:05.960772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:05.961088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:05.961355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:05.968788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:05.973596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:06.021548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 887/1773] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:05 2022] GET /identity/v3/roles => generated 1420 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:06.023207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:06.031063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:06.134573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:06.136655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:06.136960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:06.137244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:06.144574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:06.183490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:06.293221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09342c88-e5f9-4116-b2c4-9cbc02480b9a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:06.308498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b476660f-8fcb-4c9e-b68d-dc6993712ee0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:06.309521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476660f-8fcb-4c9e-b68d-dc6993712ee0 service swift] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:06.310019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476660f-8fcb-4c9e-b68d-dc6993712ee0 service swift] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:06.310504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476660f-8fcb-4c9e-b68d-dc6993712ee0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:06.312516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b476660f-8fcb-4c9e-b68d-dc6993712ee0 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:06.395772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:06.405947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:06.437596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 887/1774] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:05 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 795 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:06.452005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:06.491261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476660f-8fcb-4c9e-b68d-dc6993712ee0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:06.491608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c18d8e-7fc6-40b6-9f67-34e6051209b7 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:06.492744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476660f-8fcb-4c9e-b68d-dc6993712ee0 service swift] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:06.522298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfbb080-f6ce-45bc-aafb-c7dfa8e1a5d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:06.525236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfbb080-f6ce-45bc-aafb-c7dfa8e1a5d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:06.530157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfbb080-f6ce-45bc-aafb-c7dfa8e1a5d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:06.531944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfbb080-f6ce-45bc-aafb-c7dfa8e1a5d9 admin admin] PATH_INFO: `/v3/projects/0e317b3e30d244afabeccc23c7933dd3/users/c8310a4d63774b5aad11a43165320c97/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:06.537200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 888/1775] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:18:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 515 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:18:06.540652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:06.540961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:06.541324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:06.541771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfbb080-f6ce-45bc-aafb-c7dfa8e1a5d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e317b3e30d244afabeccc23c7933dd3, user_id=c8310a4d63774b5aad11a43165320c97, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:06.544820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfbb080-f6ce-45bc-aafb-c7dfa8e1a5d9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:06.597077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] MFA Rules not processed for user `3caf1cc366434d418c474d7a1879faa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:06.601869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 888/1776] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:06 2022] PUT /identity/v3/projects/0e317b3e30d244afabeccc23c7933dd3/users/c8310a4d63774b5aad11a43165320c97/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:06.602307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:06.621542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:06.622206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:06.625955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:06.717799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] MFA Rules not processed for user `c8310a4d63774b5aad11a43165320c97`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:06.721866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:06.743571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:06.876575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 889/1777] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:06.954105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 889/1778] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:06 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:06.988393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:06.990566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:07.251298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a4116-cc37-4f71-b714-397272fdd5d8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:07.267745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c0f3fb-9a5e-4231-8a9d-059dfb07d083 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:07.268495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0f3fb-9a5e-4231-8a9d-059dfb07d083 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:07.270376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0f3fb-9a5e-4231-8a9d-059dfb07d083 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:07.270679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0f3fb-9a5e-4231-8a9d-059dfb07d083 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:07.311644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c0f3fb-9a5e-4231-8a9d-059dfb07d083 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:07.313490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c0f3fb-9a5e-4231-8a9d-059dfb07d083 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:07.396259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 890/1779] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:06 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:07.850664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:07.852171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:07.869448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed19f15a-87e5-48e3-9b5e-b42f47dfdca8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:07.892756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:07.893353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:07.893625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:07.893896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:07.895942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:07.966950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:07.967592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:07.967592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:08.022314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] MFA Rules not processed for user `16aab142c0a7445483caf19178d56062`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:08.025630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:08.045280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:08.050127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:08.051653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:08.072569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 890/1780] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:07 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:08.226982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 891/1781] 10.222.0.25 () {58 vars in 1015 bytes} [Sat Oct 1 04:18:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:08.246856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:08.248342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:08.335538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55daf94d-51d5-42ef-9eab-555486220c7f service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:08.348419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289b699d-c61f-47f2-aecf-811b6ed608e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:08.351595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b699d-c61f-47f2-aecf-811b6ed608e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:08.352067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b699d-c61f-47f2-aecf-811b6ed608e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:08.352504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b699d-c61f-47f2-aecf-811b6ed608e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:08.373130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289b699d-c61f-47f2-aecf-811b6ed608e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:08.379644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289b699d-c61f-47f2-aecf-811b6ed608e9 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:08.421793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 891/1782] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:08.990936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:08.993548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:09.252442 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cccfd4-5c14-4829-a5f3-65e7019ae635 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:09.263686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:09.264353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:09.264637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:09.264917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] PATH_INFO: `/v3/projects/ffbe465f13f04c19b58b10aeb02886a8` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:09.266228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffbe465f13f04c19b58b10aeb02886a8)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:09.268160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:09.428100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffbe465f13f04c19b58b10aeb02886a8', 'request_id': 'req-16de3be9-ba83-43fa-945f-0eb9576b2e2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:09.478446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffbe465f13f04c19b58b10aeb02886a8', 'request_id': 'req-16de3be9-ba83-43fa-945f-0eb9576b2e2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:09.497002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffbe465f13f04c19b58b10aeb02886a8', 'request_id': 'req-16de3be9-ba83-43fa-945f-0eb9576b2e2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:09.519530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] The token cache is being invalidate because project ffbe465f13f04c19b58b10aeb02886a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:09.519831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:09.526994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-16de3be9-ba83-43fa-945f-0eb9576b2e2d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:09.535465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 892/1783] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:08 2022] DELETE /identity/v3/projects/ffbe465f13f04c19b58b10aeb02886a8 => generated 0 bytes in 541 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:09.538275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c150ab-47ef-4e2f-b3bd-34fb3c972105 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:09.538776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c150ab-47ef-4e2f-b3bd-34fb3c972105 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:09.539000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c150ab-47ef-4e2f-b3bd-34fb3c972105 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:09.615254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-01c150ab-47ef-4e2f-b3bd-34fb3c972105 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:09.616733 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-01c150ab-47ef-4e2f-b3bd-34fb3c972105 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:09.894737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2368cf-45fd-48ad-99c0-202464192890 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:09.895025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2368cf-45fd-48ad-99c0-202464192890 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:09.895455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2368cf-45fd-48ad-99c0-202464192890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:09.904259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 892/1784] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:09.997999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ee2368cf-45fd-48ad-99c0-202464192890 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:10.002830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ee2368cf-45fd-48ad-99c0-202464192890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:10.050649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 893/1785] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:10.051431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c150ab-47ef-4e2f-b3bd-34fb3c972105 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:10.075009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c150ab-47ef-4e2f-b3bd-34fb3c972105 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:10.105261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df8f03b-7f1f-4f0b-9bc8-622aee35979c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:10.105912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df8f03b-7f1f-4f0b-9bc8-622aee35979c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:10.108454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df8f03b-7f1f-4f0b-9bc8-622aee35979c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:10.114517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df8f03b-7f1f-4f0b-9bc8-622aee35979c admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:10.115287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df8f03b-7f1f-4f0b-9bc8-622aee35979c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:10.115610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df8f03b-7f1f-4f0b-9bc8-622aee35979c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:10.117319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df8f03b-7f1f-4f0b-9bc8-622aee35979c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:10.217747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 893/1786] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:18:10.223294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2368cf-45fd-48ad-99c0-202464192890 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:10.240163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2368cf-45fd-48ad-99c0-202464192890 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:10.262302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8eb600-b707-4f72-8ab7-46008e87fbc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:10.263015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8eb600-b707-4f72-8ab7-46008e87fbc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:10.263199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8eb600-b707-4f72-8ab7-46008e87fbc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:10.263469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8eb600-b707-4f72-8ab7-46008e87fbc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:10.264110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8eb600-b707-4f72-8ab7-46008e87fbc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:10.272670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8eb600-b707-4f72-8ab7-46008e87fbc4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:10.320711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a4965-eecc-48cb-b908-fd4cf4cdec49 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:10.321557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a4965-eecc-48cb-b908-fd4cf4cdec49 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:10.322181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a4965-eecc-48cb-b908-fd4cf4cdec49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:10.411075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8f5a4965-eecc-48cb-b908-fd4cf4cdec49 None None] MFA Rules not processed for user `ccd631925a0a4f5ca8be5d3915778487`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:10.413640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8f5a4965-eecc-48cb-b908-fd4cf4cdec49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:10.426057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 894/1787] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:10 2022] POST /identity/v3/projects => generated 369 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:10.426057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363821c9-99e1-42a4-8bdd-a9c6fd24b80e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:10.426057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363821c9-99e1-42a4-8bdd-a9c6fd24b80e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:10.426057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363821c9-99e1-42a4-8bdd-a9c6fd24b80e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:10.443659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:10.444013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 895/1788] 10.222.0.26 () {54 vars in 912 bytes} [Sat Oct 1 04:18:10 2022] GET /identity => generated 270 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:18:10.445043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363821c9-99e1-42a4-8bdd-a9c6fd24b80e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:10.575594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:10.624209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 894/1789] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:10 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:10.632350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb1c877-ec18-4b21-a5ff-7055765f54a3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:10.633572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb1c877-ec18-4b21-a5ff-7055765f54a3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:10.634028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb1c877-ec18-4b21-a5ff-7055765f54a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:10.716251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ddb1c877-ec18-4b21-a5ff-7055765f54a3 None None] MFA Rules not processed for user `16aab142c0a7445483caf19178d56062`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:10.722041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ddb1c877-ec18-4b21-a5ff-7055765f54a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:10.823640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363821c9-99e1-42a4-8bdd-a9c6fd24b80e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:10.849520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:10.850569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:10.850979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:10.851342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:10.852183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:10.859016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:10.944016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 896/1790] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:10 2022] POST /identity/v3/users => generated 285 bytes in 499 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:10.958786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:10.986468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc88cba-e3d5-4948-98df-58a458520521 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:11.002577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 895/1791] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:18:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:11.004243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d8d055-a296-484e-a86c-9960d8eeac9e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:11.004664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d8d055-a296-484e-a86c-9960d8eeac9e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:11.014942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d8d055-a296-484e-a86c-9960d8eeac9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:11.016989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:11.017957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:11.018256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:11.018581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:11.038813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:11.054601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:11.089333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 897/1792] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:10 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:11.094926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:11.101970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:11.108611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-70d8d055-a296-484e-a86c-9960d8eeac9e None None] MFA Rules not processed for user `7014a1defd81461787318bc27d56b9aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:11.110180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-70d8d055-a296-484e-a86c-9960d8eeac9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:11.164189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040ad061-2214-49ac-830a-ccaf6bff8c19 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:11.184630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:11.195543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:11.196326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:11.197289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:11.225407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:11.226794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:11.265191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 898/1793] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:11.266729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:11.309956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19874ba2-8d0d-40ea-b3dd-af50df18fad5 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:11.338103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1660138f-441f-42e2-a3e5-c642122c5dc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:11.338781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1660138f-441f-42e2-a3e5-c642122c5dc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:11.339168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1660138f-441f-42e2-a3e5-c642122c5dc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:11.339485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1660138f-441f-42e2-a3e5-c642122c5dc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:11.345819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1660138f-441f-42e2-a3e5-c642122c5dc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:11.353831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 896/1794] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:11 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:11.364377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1660138f-441f-42e2-a3e5-c642122c5dc5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:11.406197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 899/1795] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:11 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:11.415272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d8d055-a296-484e-a86c-9960d8eeac9e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:11.428356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d8d055-a296-484e-a86c-9960d8eeac9e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:11.487794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121ece0c-2afd-4480-8b4e-9b2654c1b927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:11.553621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ece0c-2afd-4480-8b4e-9b2654c1b927 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:11.561884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ece0c-2afd-4480-8b4e-9b2654c1b927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:11.566614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ece0c-2afd-4480-8b4e-9b2654c1b927 admin admin] PATH_INFO: `/v3/projects/3473645d8cad4d32b8c9cd186c83038f/users/370ddd74b72e4a6a9251d702e084713e/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:11.681960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121ece0c-2afd-4480-8b4e-9b2654c1b927 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3473645d8cad4d32b8c9cd186c83038f, user_id=370ddd74b72e4a6a9251d702e084713e, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:11.695170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121ece0c-2afd-4480-8b4e-9b2654c1b927 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:11.772964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 897/1796] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:11 2022] PUT /identity/v3/projects/3473645d8cad4d32b8c9cd186c83038f/users/370ddd74b72e4a6a9251d702e084713e/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:11.778561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e5019-3024-4252-b8ff-bc9a557b447c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:11.779780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e5019-3024-4252-b8ff-bc9a557b447c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:11.780239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e5019-3024-4252-b8ff-bc9a557b447c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:11.844644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-585e5019-3024-4252-b8ff-bc9a557b447c None None] MFA Rules not processed for user `370ddd74b72e4a6a9251d702e084713e`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:11.849395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-585e5019-3024-4252-b8ff-bc9a557b447c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:12.121491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa3c3f-b669-49b1-9064-422b4c3b4c58 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:12.121806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa3c3f-b669-49b1-9064-422b4c3b4c58 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:12.122169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa3c3f-b669-49b1-9064-422b4c3b4c58 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:12.125097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 898/1797] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:18:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:18:12.131138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 900/1798] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:11 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:12.133431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa3c3f-b669-49b1-9064-422b4c3b4c58 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:12.143384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea090c0-2886-466f-8a43-4a3c359bd900 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:12.143712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea090c0-2886-466f-8a43-4a3c359bd900 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:12.148417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea090c0-2886-466f-8a43-4a3c359bd900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:12.319759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-cea090c0-2886-466f-8a43-4a3c359bd900 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:12.329767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cea090c0-2886-466f-8a43-4a3c359bd900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:12.480103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:12.537945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa3c3f-b669-49b1-9064-422b4c3b4c58 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:12.571581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:12.573680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:12.577063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:12.577571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:12.579905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:12.604159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:12.605070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:12.645098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 901/1799] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:18:12 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 509 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:12.720876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 899/1800] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:12 2022] POST /identity/v3/projects => generated 369 bytes in 586 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:12.723303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea090c0-2886-466f-8a43-4a3c359bd900 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:12.742215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea090c0-2886-466f-8a43-4a3c359bd900 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:12.925984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:12.926713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:12.927090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:12.927515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:12.928271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:12.930764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:12.962324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 902/1801] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:12 2022] POST /identity/v3/users => generated 291 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:12.965943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:12.995445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6cb059-8367-42a7-af32-b36f58835925 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:13.012629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:13.013431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:13.013709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:13.014712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:13.016120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:13.017756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:13.033921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:13.034074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 900/1802] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:12 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:13.046146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafcb168-3978-4a22-8ff6-ccf28bf66d80 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:13.054542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:13.055521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:13.055894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:13.056351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] PATH_INFO: `/v3/projects/190cf5114c3c49a296a59fa5809ab2ce/users/df8ac79486fd49148354397212dfebcf/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:13.076723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] RBAC: Authorizing `identity:create_grant(project_id=190cf5114c3c49a296a59fa5809ab2ce, user_id=df8ac79486fd49148354397212dfebcf, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:13.080197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:13.101710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 903/1803] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:13 2022] PUT /identity/v3/projects/190cf5114c3c49a296a59fa5809ab2ce/users/df8ac79486fd49148354397212dfebcf/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:13.105883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:13.123244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d496ee14-9682-448b-ba4a-cb04fec579ec admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:13.346893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb7c599-decc-4c4f-a227-c071de26343e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:13.347955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb7c599-decc-4c4f-a227-c071de26343e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:13.348554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb7c599-decc-4c4f-a227-c071de26343e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:13.348856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb7c599-decc-4c4f-a227-c071de26343e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:13.349773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb7c599-decc-4c4f-a227-c071de26343e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:13.351507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb7c599-decc-4c4f-a227-c071de26343e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:13.370019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 901/1804] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:13 2022] GET /identity/v3/roles => generated 1420 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:13.371236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:13.405539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9e5c4-defa-4826-9756-2e7ada9c3949 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:13.449273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:13.450219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:13.451250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:13.451250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] PATH_INFO: `/v3/projects/190cf5114c3c49a296a59fa5809ab2ce/users/df8ac79486fd49148354397212dfebcf/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:13.453761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] RBAC: Authorizing `identity:create_grant(project_id=190cf5114c3c49a296a59fa5809ab2ce, user_id=df8ac79486fd49148354397212dfebcf, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:13.457575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:13.501037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 904/1805] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:13 2022] PUT /identity/v3/projects/190cf5114c3c49a296a59fa5809ab2ce/users/df8ac79486fd49148354397212dfebcf/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:13.504254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:13.504781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:13.505192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:13.584450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] MFA Rules not processed for user `df8ac79486fd49148354397212dfebcf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:13.595572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:13.816219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 902/1806] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:13 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:13.832812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:13.835842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:14.000647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e58d47-bc2f-407b-b4bf-6b1629a7ccac admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:14.012430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:14.013114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:14.013477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:14.013654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:14.039287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:14.040912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:14.073951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 905/1807] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:13 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:14.169104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:14.177840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:14.237943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aff59c8-f9ab-45f2-8775-612ebef5abf5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:14.244934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:14.247696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:14.250773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:14.251236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:14.397464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:14.477307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:14.480406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:14.543390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 903/1808] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:14 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:14.741196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:14.743599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:14.886563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:14.982042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bd23d7-5ee1-4f38-b469-3bf9157c4d04 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:15.007502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf0d3c-f5b6-43b9-9977-c5f5e88bac47 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:15.011038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf0d3c-f5b6-43b9-9977-c5f5e88bac47 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:15.017434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf0d3c-f5b6-43b9-9977-c5f5e88bac47 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:15.021700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf0d3c-f5b6-43b9-9977-c5f5e88bac47 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:15.077223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf0d3c-f5b6-43b9-9977-c5f5e88bac47 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:15.079883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf0d3c-f5b6-43b9-9977-c5f5e88bac47 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:15.130093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 906/1809] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:19.571234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:19.573314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:19.720418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a6854-79a4-4a6d-8c3b-2ffd6014e005 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:19.726184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:19.727694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:19.728058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:19.728331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] PATH_INFO: `/v3/users/e7245941dcba4ad7adcc9818fcf2a790` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:19.729569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7245941dcba4ad7adcc9818fcf2a790)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:19.731520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:19.792853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7245941dcba4ad7adcc9818fcf2a790'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:19.805770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7245941dcba4ad7adcc9818fcf2a790'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:19.806711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7245941dcba4ad7adcc9818fcf2a790'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:19.808404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7245941dcba4ad7adcc9818fcf2a790'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:19.817872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7245941dcba4ad7adcc9818fcf2a790'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:19.844766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 904/1810] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:19 2022] DELETE /identity/v3/users/e7245941dcba4ad7adcc9818fcf2a790 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:19.849787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:19.849787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:19.849787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:19.871063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:19.872249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:19.950983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 907/1811] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:19.965353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:19.965353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.004273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:20.005851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.021861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23720e4b-e103-4230-9d7b-2bf142063090 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.027728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.028430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.028715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.029024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] PATH_INFO: `/v3/users/370ddd74b72e4a6a9251d702e084713e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.030464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=370ddd74b72e4a6a9251d702e084713e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.032895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.041045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2bb21-9f09-4f18-88cc-19c46194cc45 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.044892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b994f0-a432-4dd2-aa49-823cfbab4479 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.045550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b994f0-a432-4dd2-aa49-823cfbab4479 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.045904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b994f0-a432-4dd2-aa49-823cfbab4479 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.046325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b994f0-a432-4dd2-aa49-823cfbab4479 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.057846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b994f0-a432-4dd2-aa49-823cfbab4479 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.059389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b994f0-a432-4dd2-aa49-823cfbab4479 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.072843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 905/1812] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:20.105518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '370ddd74b72e4a6a9251d702e084713e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.107104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '370ddd74b72e4a6a9251d702e084713e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.121096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '370ddd74b72e4a6a9251d702e084713e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.122011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '370ddd74b72e4a6a9251d702e084713e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.134961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '370ddd74b72e4a6a9251d702e084713e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.172749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 908/1813] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:20 2022] DELETE /identity/v3/users/370ddd74b72e4a6a9251d702e084713e => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:20.178827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.179218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.179617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.197726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:20.205978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:20.208907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.232684 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:20.237202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.330615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 906/1814] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:20.345391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:20.346804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.355658 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:20.379457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5b2e50-1339-431b-a63b-3d30ac19d3e2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.384337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.385007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.385295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.385605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] PATH_INFO: `/v3/users/28c3860615d54b5680997c6634bc920f` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.386940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28c3860615d54b5680997c6634bc920f)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.388915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.458925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131acbb0-1cab-46e8-8c8d-e7972c9716f0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.462977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e060bf-ab7e-468e-871e-8c9f4c59cd83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.463870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e060bf-ab7e-468e-871e-8c9f4c59cd83 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.464105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e060bf-ab7e-468e-871e-8c9f4c59cd83 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.464764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e060bf-ab7e-468e-871e-8c9f4c59cd83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.475594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28c3860615d54b5680997c6634bc920f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.476049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e060bf-ab7e-468e-871e-8c9f4c59cd83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.476781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28c3860615d54b5680997c6634bc920f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.477825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e060bf-ab7e-468e-871e-8c9f4c59cd83 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.486320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28c3860615d54b5680997c6634bc920f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.487278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28c3860615d54b5680997c6634bc920f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.488855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 907/1815] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:20.498909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28c3860615d54b5680997c6634bc920f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.538815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 909/1816] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:20 2022] DELETE /identity/v3/users/28c3860615d54b5680997c6634bc920f => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:20.544011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.544305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.544584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.549226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:20.550825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.571205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:20.572736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.646818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6341d42-92f8-4b7b-b246-b8fa356b6617 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.652384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.652495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 908/1817] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:20.653027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.653334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.653576 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] PATH_INFO: `/v3/projects/e75365d3c3d941fb8948ce24ba7dd2a9` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.654802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e75365d3c3d941fb8948ce24ba7dd2a9)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.657080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.660194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:20.661751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.713328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e75365d3c3d941fb8948ce24ba7dd2a9', 'request_id': 'req-2094c51c-7fdb-42fd-b570-35ddc01da232'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.720646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e75365d3c3d941fb8948ce24ba7dd2a9', 'request_id': 'req-2094c51c-7fdb-42fd-b570-35ddc01da232'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.733406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e75365d3c3d941fb8948ce24ba7dd2a9', 'request_id': 'req-2094c51c-7fdb-42fd-b570-35ddc01da232'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.735051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25028fc9-dbdb-4271-aae2-0e707335ff82 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.741535 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:20.742409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:20.748387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] The token cache is being invalidate because project e75365d3c3d941fb8948ce24ba7dd2a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:20.748837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.750694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.752813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 910/1818] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:20 2022] DELETE /identity/v3/projects/e75365d3c3d941fb8948ce24ba7dd2a9 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:20.756569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:20.758728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.806485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.807314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.807649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.807951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.810287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.831673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2094c51c-7fdb-42fd-b570-35ddc01da232 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.837602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.838231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.838538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.838812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] PATH_INFO: `/v3/users/c8310a4d63774b5aad11a43165320c97` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.839969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8310a4d63774b5aad11a43165320c97)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.842056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.894838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.896332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.908179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 909/1819] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:20.909003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:20.910431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:20.917619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8310a4d63774b5aad11a43165320c97'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.918538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8310a4d63774b5aad11a43165320c97'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.922587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b85d8b2-ba8c-4413-891b-f0ae81aef199 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:20.926805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8310a4d63774b5aad11a43165320c97'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.927572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:20.927929 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8310a4d63774b5aad11a43165320c97'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.928212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:20.928526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:20.928829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] PATH_INFO: `/v3/projects/3473645d8cad4d32b8c9cd186c83038f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:20.930382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3473645d8cad4d32b8c9cd186c83038f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:20.932414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:20.936883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8310a4d63774b5aad11a43165320c97'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.964586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 911/1820] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:20 2022] DELETE /identity/v3/users/c8310a4d63774b5aad11a43165320c97 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:20.990333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3473645d8cad4d32b8c9cd186c83038f', 'request_id': 'req-1618977c-ab76-4a23-9d24-d32fb478d210'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:20.999074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3473645d8cad4d32b8c9cd186c83038f', 'request_id': 'req-1618977c-ab76-4a23-9d24-d32fb478d210'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.007399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3473645d8cad4d32b8c9cd186c83038f', 'request_id': 'req-1618977c-ab76-4a23-9d24-d32fb478d210'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.021472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] The token cache is being invalidate because project 3473645d8cad4d32b8c9cd186c83038f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:21.021769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.022447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.024006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 910/1821] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:20 2022] DELETE /identity/v3/projects/3473645d8cad4d32b8c9cd186c83038f => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:21.028564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:21.030189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:21.102401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdd1eb3-be36-4e2e-b8c9-58cc57e6a767 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:21.122980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:21.165176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:21.165840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:21.166118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:21.166391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] PATH_INFO: `/v3/users/df8ac79486fd49148354397212dfebcf` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:21.167521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df8ac79486fd49148354397212dfebcf)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:21.169480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:21.220722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df8ac79486fd49148354397212dfebcf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.222042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df8ac79486fd49148354397212dfebcf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.229741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df8ac79486fd49148354397212dfebcf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.230513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df8ac79486fd49148354397212dfebcf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.239814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df8ac79486fd49148354397212dfebcf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.278238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 912/1822] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:21 2022] DELETE /identity/v3/users/df8ac79486fd49148354397212dfebcf => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:21.348994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:21.351110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:21.365766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618977c-ab76-4a23-9d24-d32fb478d210 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:21.416752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:21.454180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:21.455156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:21.455657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:21.456090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] PATH_INFO: `/v3/projects/5f0e67b69acc4cb5844b183fd3176880` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:21.457531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f0e67b69acc4cb5844b183fd3176880)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:21.459929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:21.535563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f0e67b69acc4cb5844b183fd3176880', 'request_id': 'req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.543654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f0e67b69acc4cb5844b183fd3176880', 'request_id': 'req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.553256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f0e67b69acc4cb5844b183fd3176880', 'request_id': 'req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.565479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] The token cache is being invalidate because project 5f0e67b69acc4cb5844b183fd3176880 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:21.565778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.566515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.568115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 911/1823] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:21 2022] DELETE /identity/v3/projects/5f0e67b69acc4cb5844b183fd3176880 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:21.572507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:21.574328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:21.592733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc73b2e-1652-4f7e-bb17-3a1f1a696548 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:21.641530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:21.643575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:21.692044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:21.692793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:21.693725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:21.694072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] PATH_INFO: `/v3/users/6b5dc5392a46420cbe4586138c3fb573` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:21.695321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b5dc5392a46420cbe4586138c3fb573)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:21.699727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:21.738628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3aadda-b8ff-46a4-9b59-ba7c77d3590f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:21.747379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:21.748486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:21.749134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:21.751302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] PATH_INFO: `/v3/projects/0e317b3e30d244afabeccc23c7933dd3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:21.754014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e317b3e30d244afabeccc23c7933dd3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:21.758382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:21.772267 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:21.773862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b5dc5392a46420cbe4586138c3fb573'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.775937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b5dc5392a46420cbe4586138c3fb573'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.795940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b5dc5392a46420cbe4586138c3fb573'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.807679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b5dc5392a46420cbe4586138c3fb573'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.820465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b5dc5392a46420cbe4586138c3fb573'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.832786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e317b3e30d244afabeccc23c7933dd3', 'request_id': 'req-260ee3af-a874-4e17-9c2c-0edae546e2cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.842115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 913/1824] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:21 2022] DELETE /identity/v3/users/6b5dc5392a46420cbe4586138c3fb573 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:21.845189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e317b3e30d244afabeccc23c7933dd3', 'request_id': 'req-260ee3af-a874-4e17-9c2c-0edae546e2cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.853025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e317b3e30d244afabeccc23c7933dd3', 'request_id': 'req-260ee3af-a874-4e17-9c2c-0edae546e2cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.862626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:21.866960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] The token cache is being invalidate because project 0e317b3e30d244afabeccc23c7933dd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:21.867486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.868292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-260ee3af-a874-4e17-9c2c-0edae546e2cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:21.870286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 912/1825] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:21 2022] DELETE /identity/v3/projects/0e317b3e30d244afabeccc23c7933dd3 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:21.877068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2680c93d-779d-4f32-bddc-2afb44a4c3ff None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:21.877565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2680c93d-779d-4f32-bddc-2afb44a4c3ff None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:21.878421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2680c93d-779d-4f32-bddc-2afb44a4c3ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:21.879305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:21.881405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:21.910706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-2680c93d-779d-4f32-bddc-2afb44a4c3ff None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:21.912130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2680c93d-779d-4f32-bddc-2afb44a4c3ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:21.978331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704fef02-f5e1-445c-8e4e-f3bdd429f7c4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:21.985523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:21.986255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:21.986504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:21.987347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] PATH_INFO: `/v3/projects/190cf5114c3c49a296a59fa5809ab2ce` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:21.987931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=190cf5114c3c49a296a59fa5809ab2ce)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:21.988882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 913/1826] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:21.990433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:21.996804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bc78c7-2679-49b2-b881-df91adda3bb9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:21.997129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bc78c7-2679-49b2-b881-df91adda3bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:21.997406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bc78c7-2679-49b2-b881-df91adda3bb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.024161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-88bc78c7-2679-49b2-b881-df91adda3bb9 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:22.025507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-88bc78c7-2679-49b2-b881-df91adda3bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.038017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 914/1827] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:22.038937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bc78c7-2679-49b2-b881-df91adda3bb9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.057580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '190cf5114c3c49a296a59fa5809ab2ce', 'request_id': 'req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.058494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bc78c7-2679-49b2-b881-df91adda3bb9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.064445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '190cf5114c3c49a296a59fa5809ab2ce', 'request_id': 'req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.074293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '190cf5114c3c49a296a59fa5809ab2ce', 'request_id': 'req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.090709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] The token cache is being invalidate because project 190cf5114c3c49a296a59fa5809ab2ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:22.091019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.091693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-0b2df0d4-e4b7-4135-b524-4b1a4d02c4c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.094232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 914/1828] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:21 2022] DELETE /identity/v3/projects/190cf5114c3c49a296a59fa5809ab2ce => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:22.098915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20fdb6-f0c1-4f56-9a9f-4f3e603bd25b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.099200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20fdb6-f0c1-4f56-9a9f-4f3e603bd25b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.099480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20fdb6-f0c1-4f56-9a9f-4f3e603bd25b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.124178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9b20fdb6-f0c1-4f56-9a9f-4f3e603bd25b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:22.126319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9b20fdb6-f0c1-4f56-9a9f-4f3e603bd25b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.128131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.128810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.129075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.129407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.130088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:22.130544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.132175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.136898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 915/1829] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:22.144988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 915/1830] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:22.145360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190c6a6-ed99-406e-9f34-3b097acadb3c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.145724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190c6a6-ed99-406e-9f34-3b097acadb3c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.146064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190c6a6-ed99-406e-9f34-3b097acadb3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.149705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.151668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.164212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe11321-55d6-4042-92c0-9d3f97f32da2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.169104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.169978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.170182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.170506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.172004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.173600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-1190c6a6-ed99-406e-9f34-3b097acadb3c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:22.174102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.174993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1190c6a6-ed99-406e-9f34-3b097acadb3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.192547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 916/1831] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:22.195617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1190c6a6-ed99-406e-9f34-3b097acadb3c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.205623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 916/1832] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:22.209344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.211627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1190c6a6-ed99-406e-9f34-3b097acadb3c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.260141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:22.277135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541cbc12-6936-44ac-8929-31636156f44c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.282212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.282861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.283088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.283264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.283404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.283694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.283975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.284081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:22.284274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.284420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.284876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.285931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.286880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.298065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 917/1833] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:22.298688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.298811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:22.300464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.312100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 917/1834] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:22.312902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.315892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00e55c5-1962-465b-a13a-c1579b7555ea admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.321203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.321832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.322116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.322391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] PATH_INFO: `/v3/projects/d1f02d5f57204f339652485e863eddd4` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.323270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae79a436-97f8-402f-b65d-6e0d594609c4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.323514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1f02d5f57204f339652485e863eddd4)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.325470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.328594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.329211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.329487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.329785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.330491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.332455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.363400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 918/1835] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:22.363400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.381360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1f02d5f57204f339652485e863eddd4', 'request_id': 'req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.387678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1f02d5f57204f339652485e863eddd4', 'request_id': 'req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.396389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1f02d5f57204f339652485e863eddd4', 'request_id': 'req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.409392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] The token cache is being invalidate because project d1f02d5f57204f339652485e863eddd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:22.409685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.410346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:22.415509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 918/1836] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:22 2022] DELETE /identity/v3/projects/d1f02d5f57204f339652485e863eddd4 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:22.415509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.417355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.430052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78012ad2-35c9-4268-a3b7-bbbc58fdfbe5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.494143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4263ec-43d4-41e2-bbe0-0ebcf9c52507 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.501255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.507055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.507055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.507055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.507055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.507055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.509501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c67327-807b-4766-bc0b-ad1fc9351025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.510207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c67327-807b-4766-bc0b-ad1fc9351025 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.510565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c67327-807b-4766-bc0b-ad1fc9351025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.510913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c67327-807b-4766-bc0b-ad1fc9351025 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.511644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c67327-807b-4766-bc0b-ad1fc9351025 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.513348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c67327-807b-4766-bc0b-ad1fc9351025 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.528047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 919/1837] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:22 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:22.531930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 919/1838] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/users => generated 292 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:22.532081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea30c02c-fe02-4030-86e7-d38dd42babbd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.532081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea30c02c-fe02-4030-86e7-d38dd42babbd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.532081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea30c02c-fe02-4030-86e7-d38dd42babbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.533191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.543230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbd4e9f-f742-4a07-95e6-c7d426709015 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.548739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.549364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.549670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.549938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.550570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.552162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.552990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ea30c02c-fe02-4030-86e7-d38dd42babbd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:22.554299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea30c02c-fe02-4030-86e7-d38dd42babbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.562379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 920/1839] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:22.563163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.568201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 920/1840] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:22.568201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea30c02c-fe02-4030-86e7-d38dd42babbd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.569820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.577633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea30c02c-fe02-4030-86e7-d38dd42babbd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.582457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0682cde9-5bd5-4346-a7ed-d05167677b10 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.582647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f7648f-7348-403e-9110-8902a3b12732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.583249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f7648f-7348-403e-9110-8902a3b12732 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.583533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f7648f-7348-403e-9110-8902a3b12732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.583938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f7648f-7348-403e-9110-8902a3b12732 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.584590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f7648f-7348-403e-9110-8902a3b12732 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.586212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f7648f-7348-403e-9110-8902a3b12732 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.587353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.587943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.588215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.588487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.589144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.590698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.600985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 921/1841] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:22.600985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc310078-cd6b-439e-862b-50d668c8c4dc None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.600985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc310078-cd6b-439e-862b-50d668c8c4dc None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.600985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc310078-cd6b-439e-862b-50d668c8c4dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.606748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 921/1842] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:22 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:22.606748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.612593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad99a6b-6f6d-4872-8f01-9644401abab0 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.617129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.617773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.618051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.618323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] PATH_INFO: `/v3/projects/19c266c000864c388ebe2e5c7346d40f/users/6a6875735dfe460ea55b05b1357e26c6/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.621746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cc310078-cd6b-439e-862b-50d668c8c4dc None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:22.623011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cc310078-cd6b-439e-862b-50d668c8c4dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.633915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 922/1843] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:22.634750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc310078-cd6b-439e-862b-50d668c8c4dc None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.639143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19c266c000864c388ebe2e5c7346d40f, user_id=6a6875735dfe460ea55b05b1357e26c6, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.642611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.645680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc310078-cd6b-439e-862b-50d668c8c4dc None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.652209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e719a9-b9f8-4d9a-9c31-c4dc071c75b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.653022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:22.653981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e719a9-b9f8-4d9a-9c31-c4dc071c75b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.654634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e719a9-b9f8-4d9a-9c31-c4dc071c75b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.655716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e719a9-b9f8-4d9a-9c31-c4dc071c75b6 admin admin] PATH_INFO: `/v3/projects/7efac16f6e8d44d69db812ba60250a97/users/05aba8dccc54435abc49c4d04dd2d150/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.658898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 922/1844] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:22 2022] PUT /identity/v3/projects/19c266c000864c388ebe2e5c7346d40f/users/6a6875735dfe460ea55b05b1357e26c6/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:22.660000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.676531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e719a9-b9f8-4d9a-9c31-c4dc071c75b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7efac16f6e8d44d69db812ba60250a97, user_id=05aba8dccc54435abc49c4d04dd2d150, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.679630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e719a9-b9f8-4d9a-9c31-c4dc071c75b6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.695330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 923/1845] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:22 2022] PUT /identity/v3/projects/7efac16f6e8d44d69db812ba60250a97/users/05aba8dccc54435abc49c4d04dd2d150/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:22.704819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1128aa-aed2-4f77-835b-b998ec0eee9d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.704819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1128aa-aed2-4f77-835b-b998ec0eee9d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.704819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1128aa-aed2-4f77-835b-b998ec0eee9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.731451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4b1128aa-aed2-4f77-835b-b998ec0eee9d None None] MFA Rules not processed for user `6a6875735dfe460ea55b05b1357e26c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:22.733322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4b1128aa-aed2-4f77-835b-b998ec0eee9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.743088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bbc0b0-6db1-47c9-a551-b207b82cd660 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.751818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:22.803722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 924/1846] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:22.803722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaaafcf-1abc-4a10-ba8a-5f2b810725ef None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.803722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaaafcf-1abc-4a10-ba8a-5f2b810725ef None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.803722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaaafcf-1abc-4a10-ba8a-5f2b810725ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.814854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.815505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.815793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.816138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.816709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:22.817050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.818719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.834994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 923/1847] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:22.835874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:22.846698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b1b233-515d-41e5-83f6-53101bd1ed82 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:22.853180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:22.853967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:22.854449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:22.854906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:22.855847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:22.858111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:22.861432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-bfaaafcf-1abc-4a10-ba8a-5f2b810725ef None None] MFA Rules not processed for user `05aba8dccc54435abc49c4d04dd2d150`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:22.866708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bfaaafcf-1abc-4a10-ba8a-5f2b810725ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:22.917477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 924/1848] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/projects => generated 385 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:22.918992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.042585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 925/1849] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:23.043878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaaafcf-1abc-4a10-ba8a-5f2b810725ef None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.045887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9256cf-5659-421d-b7ed-b4491fa47db9 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.065668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.066307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.066617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.066900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.067578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.073819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.090946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaaafcf-1abc-4a10-ba8a-5f2b810725ef None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.096836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.097474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.097865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.098139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.098767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.100832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.131464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 926/1850] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:23 2022] POST /identity/v3/users => generated 299 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:23.131657 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 925/1851] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:22 2022] POST /identity/v3/projects => generated 369 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:23.136755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.136951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.142162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:23.153241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56353a7-ba6d-422d-87e2-4bc4d9cd517a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.234860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4e5bc4-8bb6-4263-b6a5-4983155933f5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.243176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.245091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.245953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.247723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.248181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.251329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.253802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.254656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.255552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.256168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.257760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.259743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.279541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 927/1852] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:23.279541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.287720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84824abb-247c-401a-bbdd-c14ada5d842a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.287889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 926/1853] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:23 2022] POST /identity/v3/projects => generated 383 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:23.291800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.311541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:23.347726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.348341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.348735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.349078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.349734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.351649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.355416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ca1070-deb8-4978-8345-25a3da544dc0 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.360223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.360906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.361225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.361504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] PATH_INFO: `/v3/projects/0132cdfcf4ee4193b2d4b5951535faa7/users/a836f458474340d9b03e611dc0591a7f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.373576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 928/1854] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:23 2022] POST /identity/v3/users => generated 285 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:23.374899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.382992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=0132cdfcf4ee4193b2d4b5951535faa7, user_id=a836f458474340d9b03e611dc0591a7f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.385674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.386970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e87ec86-c349-4ef5-878d-63362f8f430e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.392658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.393247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.393528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.393802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.394432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.396336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.400100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 927/1855] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:23 2022] PUT /identity/v3/projects/0132cdfcf4ee4193b2d4b5951535faa7/users/a836f458474340d9b03e611dc0591a7f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:23.400843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.418391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 929/1856] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:23 2022] POST /identity/v3/users => generated 298 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:23.419455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.431382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d869852b-15ad-4bff-b7ed-638ba569122d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.465491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cfd1b7-efe8-47aa-80d8-3727e1efe9ff admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.470338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.470964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.471248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.471548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.472188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.473809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.483450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 928/1857] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:23.484534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.491078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.491724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.492018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.492289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.492943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.494151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cb8350-4fa1-423c-92e0-69447e3c87d5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.494524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.498743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.499330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.500147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.500421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.501104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.505363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 930/1858] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:23.508470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.509299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:23.509480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.520586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1c3fc4-6cd7-411c-af6f-8ac229f9702d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.521187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 929/1859] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:23.522693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.525960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.526869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.527155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.527432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.528106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.530303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.533774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907edbe0-2cb3-4c46-9f72-ace152fa2bce admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.538729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.539457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.539830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.540111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] PATH_INFO: `/v3/projects/0132cdfcf4ee4193b2d4b5951535faa7/users/a836f458474340d9b03e611dc0591a7f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.540732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 931/1860] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:23.541714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.545193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0132cdfcf4ee4193b2d4b5951535faa7, user_id=a836f458474340d9b03e611dc0591a7f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.548355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.555827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fdeaa-2de2-4ded-8e94-f79ae4f01553 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.560083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 930/1861] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:23 2022] PUT /identity/v3/projects/0132cdfcf4ee4193b2d4b5951535faa7/users/a836f458474340d9b03e611dc0591a7f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:23.565816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.633335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e034910-992c-4972-8cb7-5410a6e1283f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.634009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e034910-992c-4972-8cb7-5410a6e1283f admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.634289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e034910-992c-4972-8cb7-5410a6e1283f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.634566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e034910-992c-4972-8cb7-5410a6e1283f admin admin] PATH_INFO: `/v3/projects/4e3b25808ff14c8b9aa65c8a2e3b2b19/users/ef18ed21d62a4b48824245712307ce8c/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.639754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eee54f-8cf5-4a1e-9764-7135e31f5487 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.644827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.645475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.645757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.646035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] PATH_INFO: `/v3/projects/aa03cfd05b6a4245ab91f2173cebb615/users/c32624d142314679bdfe3027832d1b11/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.649134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e034910-992c-4972-8cb7-5410a6e1283f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e3b25808ff14c8b9aa65c8a2e3b2b19, user_id=ef18ed21d62a4b48824245712307ce8c, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.652340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e034910-992c-4972-8cb7-5410a6e1283f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.661831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa03cfd05b6a4245ab91f2173cebb615, user_id=c32624d142314679bdfe3027832d1b11, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.664554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.665504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 932/1862] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:23 2022] PUT /identity/v3/projects/4e3b25808ff14c8b9aa65c8a2e3b2b19/users/ef18ed21d62a4b48824245712307ce8c/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:23.666868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482923a3-6a25-476e-a52d-7b0553f89be0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.667158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482923a3-6a25-476e-a52d-7b0553f89be0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.667435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482923a3-6a25-476e-a52d-7b0553f89be0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.680726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 931/1863] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:23 2022] PUT /identity/v3/projects/aa03cfd05b6a4245ab91f2173cebb615/users/c32624d142314679bdfe3027832d1b11/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:23.682626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:23.685797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.697857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-482923a3-6a25-476e-a52d-7b0553f89be0 None None] MFA Rules not processed for user `a836f458474340d9b03e611dc0591a7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:23.699065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-482923a3-6a25-476e-a52d-7b0553f89be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:23.748808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d316890a-1be2-4d39-b110-c79a1c9bcc83 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.753939 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fc3e08-e66d-463a-a3b7-e06d63e563b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.754547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc3e08-e66d-463a-a3b7-e06d63e563b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.754827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc3e08-e66d-463a-a3b7-e06d63e563b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.755099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc3e08-e66d-463a-a3b7-e06d63e563b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.755763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fc3e08-e66d-463a-a3b7-e06d63e563b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.757393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fc3e08-e66d-463a-a3b7-e06d63e563b0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.773529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 932/1864] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:23.774578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 933/1865] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:23.774875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.775168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.775509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.775770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482923a3-6a25-476e-a52d-7b0553f89be0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.794833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482923a3-6a25-476e-a52d-7b0553f89be0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.800454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.801169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.801455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.801728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] PATH_INFO: `/v3/projects/4e3b25808ff14c8b9aa65c8a2e3b2b19/users/ef18ed21d62a4b48824245712307ce8c/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.803024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:23.803690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e3b25808ff14c8b9aa65c8a2e3b2b19, user_id=ef18ed21d62a4b48824245712307ce8c, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.804276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:23.806396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.815762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 933/1866] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:23.815762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.815762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:23.825019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 934/1867] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:23 2022] PUT /identity/v3/projects/4e3b25808ff14c8b9aa65c8a2e3b2b19/users/ef18ed21d62a4b48824245712307ce8c/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:23.827053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:23.831797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:23.886123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ef85e1-fa38-410c-bca3-5f3bf4cc6790 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.896115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:23.904449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355322d2-beb5-4f48-8423-3feb702cf378 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:23.908820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ce3828-8f44-4c54-a076-33e6a9716af4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.909502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ce3828-8f44-4c54-a076-33e6a9716af4 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.909793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ce3828-8f44-4c54-a076-33e6a9716af4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.910068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ce3828-8f44-4c54-a076-33e6a9716af4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.939797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:23.940464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.940746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.941045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:23.972355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ce3828-8f44-4c54-a076-33e6a9716af4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:23.973943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ce3828-8f44-4c54-a076-33e6a9716af4 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:23.985321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 935/1868] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:23.986525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d864b126-658b-4700-8b03-308e07707424 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:23.986939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d864b126-658b-4700-8b03-308e07707424 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:23.987227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d864b126-658b-4700-8b03-308e07707424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.017981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:24.019421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:24.030526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 934/1869] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:23 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:24.046452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d864b126-658b-4700-8b03-308e07707424 None None] MFA Rules not processed for user `ef18ed21d62a4b48824245712307ce8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:24.047770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d864b126-658b-4700-8b03-308e07707424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.074700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:24.075471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:24.092095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98680bd4-38a2-4b0a-905e-df141c308f2f service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:24.098740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:24.099401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:24.099963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:24.100257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] PATH_INFO: `/v3/users/6a6875735dfe460ea55b05b1357e26c6` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.102150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a6875735dfe460ea55b05b1357e26c6)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:24.104182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:24.154771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 936/1870] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:23 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:24.165479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d864b126-658b-4700-8b03-308e07707424 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:24.170850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a6875735dfe460ea55b05b1357e26c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.173495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a6875735dfe460ea55b05b1357e26c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.187369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a6875735dfe460ea55b05b1357e26c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.188808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a6875735dfe460ea55b05b1357e26c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.194075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d864b126-658b-4700-8b03-308e07707424 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:24.200360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-756e783c-a6c3-4644-91ae-f3e076e16e64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a6875735dfe460ea55b05b1357e26c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.207610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:24.208272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:24.208527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:24.208797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.211803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.227400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 935/1871] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:24 2022] DELETE /identity/v3/users/6a6875735dfe460ea55b05b1357e26c6 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:24.233055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:24.233490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:24.233859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.239152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:24.243275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:24.259827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 937/1872] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:24 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:24.275144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:24.277163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.334704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:24.366329 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:24.367991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.371177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 936/1873] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:24.381124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:24.383330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b789a08-6b19-4377-9517-f06343db0378 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:24.384898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.449600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1136f2b-f4f5-4f37-a0fc-a06c1f512c61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:24.450276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1136f2b-f4f5-4f37-a0fc-a06c1f512c61 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:24.450564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1136f2b-f4f5-4f37-a0fc-a06c1f512c61 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:24.450936 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1136f2b-f4f5-4f37-a0fc-a06c1f512c61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.453565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d1136f2b-f4f5-4f37-a0fc-a06c1f512c61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.480760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b4b257-a562-4834-9fc2-451719a0ee69 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:24.486676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:24.488222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:24.488609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:24.491377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.506367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:24.510979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:24.524996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 937/1874] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:24.535472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1136f2b-f4f5-4f37-a0fc-a06c1f512c61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:24.536964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1136f2b-f4f5-4f37-a0fc-a06c1f512c61 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:24.545897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 938/1875] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:24.835916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:24.837605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.851113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fea13f-3869-42a9-8c05-197843d8ff7a service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:24.856226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:24.856991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:24.857370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:24.857686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] PATH_INFO: `/v3/users/05aba8dccc54435abc49c4d04dd2d150` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.859065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] RBAC: Authorizing `identity:delete_user(user_id=05aba8dccc54435abc49c4d04dd2d150)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:24.861654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:24.920356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:24.920845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:24.921726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:24.964273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05aba8dccc54435abc49c4d04dd2d150'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.965667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05aba8dccc54435abc49c4d04dd2d150'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.974711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] MFA Rules not processed for user `c32624d142314679bdfe3027832d1b11`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:24.976473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:24.981335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05aba8dccc54435abc49c4d04dd2d150'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.982188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05aba8dccc54435abc49c4d04dd2d150'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:24.999964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5586b2af-df76-45c7-a747-49736700409c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05aba8dccc54435abc49c4d04dd2d150'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.031004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 938/1876] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:24 2022] DELETE /identity/v3/users/05aba8dccc54435abc49c4d04dd2d150 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:25.035328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635328d3-eebf-494f-b3dd-090309338786 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.035692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635328d3-eebf-494f-b3dd-090309338786 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.036041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635328d3-eebf-494f-b3dd-090309338786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.059472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-635328d3-eebf-494f-b3dd-090309338786 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:25.060783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-635328d3-eebf-494f-b3dd-090309338786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.083454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:25.111114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 939/1877] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:24 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:25.115192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.116710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.145794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375ecbbb-28e6-4bc5-9139-5314132ca86f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.164232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:25.205046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 939/1878] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:25.205046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635328d3-eebf-494f-b3dd-090309338786 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.217382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-635328d3-eebf-494f-b3dd-090309338786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.230008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635328d3-eebf-494f-b3dd-090309338786 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.232469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.233116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.233397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.234210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] PATH_INFO: `/v3/projects/19c266c000864c388ebe2e5c7346d40f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.235645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19c266c000864c388ebe2e5c7346d40f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.237093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.237625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.237946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.238323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.238695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.239530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.241661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.273683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 940/1879] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:25 2022] POST /identity/v3/projects => generated 373 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:25.274446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.278022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.289533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19c266c000864c388ebe2e5c7346d40f', 'request_id': 'req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.298215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19c266c000864c388ebe2e5c7346d40f', 'request_id': 'req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.304506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19c266c000864c388ebe2e5c7346d40f', 'request_id': 'req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.315859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] The token cache is being invalidate because project 19c266c000864c388ebe2e5c7346d40f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:25.316147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.316785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.318303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 940/1880] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:25 2022] DELETE /identity/v3/projects/19c266c000864c388ebe2e5c7346d40f => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:25.321507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.321507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.340290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4768f4-1627-4d22-aee0-49448bf8f354 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.391599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70d85d8-82ad-4ef0-b4e6-62cebff5ebd0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.393242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.393855 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.394269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.394558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.396366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.397441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.398212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.398319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.398579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.399240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.401173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.412051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.413518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.425813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 941/1881] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:25 2022] POST /identity/v3/users => generated 293 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:25.425988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 941/1882] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:25.426948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.428241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.429295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.437545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:25.441129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f55ec8f-fa74-4fc2-b3ab-c75d4723972b service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.442044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c946c240-839b-494f-91e8-7dfeb229d586 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.449638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.450510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.450861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.451223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] PATH_INFO: `/v3/users/a836f458474340d9b03e611dc0591a7f` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.452122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.452549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] RBAC: Authorizing `identity:delete_user(user_id=a836f458474340d9b03e611dc0591a7f)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.452941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.453302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.453651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.454428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.455022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.457041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.469084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 942/1883] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:25 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:25.473112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.484311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92aa7073-d820-4b70-8f07-9ef5b11b6320 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.492511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.493452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.493762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.494014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] PATH_INFO: `/v3/projects/a8dc6a43c1f64c3c9d64b669e1ea93ea/users/6930512abf8d4fdcadbf059ec4c4149f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.515139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8dc6a43c1f64c3c9d64b669e1ea93ea, user_id=6930512abf8d4fdcadbf059ec4c4149f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.518777 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a836f458474340d9b03e611dc0591a7f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.519564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a836f458474340d9b03e611dc0591a7f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.520892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.531063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a836f458474340d9b03e611dc0591a7f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.532255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a836f458474340d9b03e611dc0591a7f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.542231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 943/1884] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:25 2022] PUT /identity/v3/projects/a8dc6a43c1f64c3c9d64b669e1ea93ea/users/6930512abf8d4fdcadbf059ec4c4149f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:25.546061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.548481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.551560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a836f458474340d9b03e611dc0591a7f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:25.585355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:25.601016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 942/1885] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:25 2022] DELETE /identity/v3/users/a836f458474340d9b03e611dc0591a7f => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:25.639679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd9b35a-4541-425a-948f-e6c047cda190 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.698707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.699239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.699522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.699795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.700469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.702125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.713609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 944/1886] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:25 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:18:25.717478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.727959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4331a61-d814-4332-aaa2-08fd122e0fde admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:25.732826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:25.733905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:25.733905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:25.734221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] PATH_INFO: `/v3/projects/a8dc6a43c1f64c3c9d64b669e1ea93ea/users/6930512abf8d4fdcadbf059ec4c4149f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:25.736378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8dc6a43c1f64c3c9d64b669e1ea93ea, user_id=6930512abf8d4fdcadbf059ec4c4149f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:25.739044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:25.750227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 943/1887] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:25 2022] PUT /identity/v3/projects/a8dc6a43c1f64c3c9d64b669e1ea93ea/users/6930512abf8d4fdcadbf059ec4c4149f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:25.937880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:25.941328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:25.962857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b449962-a615-4977-a691-34b86d036b46 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:26.123168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:26.127723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:26.151058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:26.152677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:26.153539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:26.155161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] PATH_INFO: `/v3/projects/7efac16f6e8d44d69db812ba60250a97` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:26.172317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7efac16f6e8d44d69db812ba60250a97)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:26.187502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:26.208458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:26.279810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7efac16f6e8d44d69db812ba60250a97', 'request_id': 'req-194d5e4d-2328-4d98-82c7-ae9e04cedc70'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.300661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7efac16f6e8d44d69db812ba60250a97', 'request_id': 'req-194d5e4d-2328-4d98-82c7-ae9e04cedc70'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.312857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7efac16f6e8d44d69db812ba60250a97', 'request_id': 'req-194d5e4d-2328-4d98-82c7-ae9e04cedc70'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.320559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8759d85c-837f-4937-bd84-4672c8b7dfbd admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:26.339951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] The token cache is being invalidate because project 7efac16f6e8d44d69db812ba60250a97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:26.341186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.341974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.345034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 945/1888] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:25 2022] DELETE /identity/v3/projects/7efac16f6e8d44d69db812ba60250a97 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:26.351722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:26.354156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:26.440082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:26.441185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:26.441537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:26.441878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] PATH_INFO: `/v3/projects/0132cdfcf4ee4193b2d4b5951535faa7` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:26.443988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0132cdfcf4ee4193b2d4b5951535faa7)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:26.446659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:26.457242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194d5e4d-2328-4d98-82c7-ae9e04cedc70 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:26.467300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:26.468668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:26.468977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:26.469317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] PATH_INFO: `/v3/users/ef18ed21d62a4b48824245712307ce8c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:26.470536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef18ed21d62a4b48824245712307ce8c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:26.473317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:26.573715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0132cdfcf4ee4193b2d4b5951535faa7', 'request_id': 'req-91956003-6e2e-4ce8-8f00-a3451251c868'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.577116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef18ed21d62a4b48824245712307ce8c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.591149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0132cdfcf4ee4193b2d4b5951535faa7', 'request_id': 'req-91956003-6e2e-4ce8-8f00-a3451251c868'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.592066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef18ed21d62a4b48824245712307ce8c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.595031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef18ed21d62a4b48824245712307ce8c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.596270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef18ed21d62a4b48824245712307ce8c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.603660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0132cdfcf4ee4193b2d4b5951535faa7', 'request_id': 'req-91956003-6e2e-4ce8-8f00-a3451251c868'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.611635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-a8e5b0a7-fdc9-4db5-9b11-02bef566bab3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef18ed21d62a4b48824245712307ce8c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.628694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] The token cache is being invalidate because project 0132cdfcf4ee4193b2d4b5951535faa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:26.630698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.634909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-91956003-6e2e-4ce8-8f00-a3451251c868 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:26.636542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 944/1889] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:26 2022] DELETE /identity/v3/projects/0132cdfcf4ee4193b2d4b5951535faa7 => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:26.642108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb56a63-4fd8-400c-976c-7f46f89a40f7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:26.642390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb56a63-4fd8-400c-976c-7f46f89a40f7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:26.642676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb56a63-4fd8-400c-976c-7f46f89a40f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:26.679072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 946/1890] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:26 2022] DELETE /identity/v3/users/ef18ed21d62a4b48824245712307ce8c => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:26.694015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-edb56a63-4fd8-400c-976c-7f46f89a40f7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:26.700649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-edb56a63-4fd8-400c-976c-7f46f89a40f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:26.729548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:26.796802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 945/1891] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:26.805168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f46bd-9c93-4e60-9df2-56ca0ce27f8d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:26.806102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f46bd-9c93-4e60-9df2-56ca0ce27f8d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:26.806102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f46bd-9c93-4e60-9df2-56ca0ce27f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:26.830155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ca0f46bd-9c93-4e60-9df2-56ca0ce27f8d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:26.832270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ca0f46bd-9c93-4e60-9df2-56ca0ce27f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:26.846684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 947/1892] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:26.852648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb56a63-4fd8-400c-976c-7f46f89a40f7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:26.869618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb56a63-4fd8-400c-976c-7f46f89a40f7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:26.877223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:26.878819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:26.879176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:26.879673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:26.880459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:26.880942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:26.884095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:26.902325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 946/1893] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:26.903162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0f46bd-9c93-4e60-9df2-56ca0ce27f8d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:26.916397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0f46bd-9c93-4e60-9df2-56ca0ce27f8d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:26.923998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:26.924646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:26.924900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:26.925327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:26.926205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:26.929161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:26.961166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:26.991183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:26.991576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 948/1894] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:26 2022] POST /identity/v3/projects => generated 379 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:27.005935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd8a92-1992-40fc-a2c9-830e2a2f7059 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.111209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.112103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.112394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.112673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.113362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:27.117783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:27.152010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:27.163224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 947/1895] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:26 2022] POST /identity/v3/users => generated 290 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:27.166535 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75c2ac3-c390-4fb0-830e-6173b5f6e022 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.175433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.176178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.176492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.176818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.178563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:27.180221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:27.197617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:27.207101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 949/1896] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:27 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:27.209460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d367acd-4336-4358-a662-6fd0ee5e7fef admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.214520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc94b7f-9cb8-40c8-90e8-3db17f5a4a0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.215226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc94b7f-9cb8-40c8-90e8-3db17f5a4a0c admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.215579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc94b7f-9cb8-40c8-90e8-3db17f5a4a0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.215924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc94b7f-9cb8-40c8-90e8-3db17f5a4a0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.216654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc94b7f-9cb8-40c8-90e8-3db17f5a4a0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:27.219038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc94b7f-9cb8-40c8-90e8-3db17f5a4a0c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:27.236707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 948/1897] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:27 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:27.237511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:27.255264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09049af9-9665-47a7-9082-d02f9d3963d8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.260840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.262654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.263171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.263718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] PATH_INFO: `/v3/projects/defbf4f78edd4a1ea496d5188a17685a/users/593547adf9f24e17ab65bc701c90b766/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.286186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=defbf4f78edd4a1ea496d5188a17685a, user_id=593547adf9f24e17ab65bc701c90b766, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:27.290252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:27.324904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 950/1898] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:27 2022] PUT /identity/v3/projects/defbf4f78edd4a1ea496d5188a17685a/users/593547adf9f24e17ab65bc701c90b766/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:27.338484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540ad18-4575-4d3c-a596-7dd877f0a7f0 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.339439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540ad18-4575-4d3c-a596-7dd877f0a7f0 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.339919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540ad18-4575-4d3c-a596-7dd877f0a7f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.397740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-3540ad18-4575-4d3c-a596-7dd877f0a7f0 None None] MFA Rules not processed for user `593547adf9f24e17ab65bc701c90b766`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:27.399703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3540ad18-4575-4d3c-a596-7dd877f0a7f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:27.488218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 949/1899] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:27.494058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:27.586616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613e577d-54c3-4e03-b84a-ac54ed2d63ae admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.602557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.603072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.603356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.603631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.604272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:27.609727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:27.658047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3540ad18-4575-4d3c-a596-7dd877f0a7f0 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:27.662863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 951/1900] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:27 2022] POST /identity/v3/projects => generated 383 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:27.674165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3540ad18-4575-4d3c-a596-7dd877f0a7f0 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.701758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:27.818187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.819328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.819620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.821201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.822472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:27.827517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:27.879783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:27.880265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 950/1901] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:27 2022] POST /identity/v3/users => generated 298 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:27.894810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ce6207-89bd-4b01-a381-d38704b631c3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.901980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60202e1d-5389-4f21-8793-28827d10cd4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.903084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60202e1d-5389-4f21-8793-28827d10cd4c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.903685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60202e1d-5389-4f21-8793-28827d10cd4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.904100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60202e1d-5389-4f21-8793-28827d10cd4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.904904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60202e1d-5389-4f21-8793-28827d10cd4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:27.907771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60202e1d-5389-4f21-8793-28827d10cd4c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:27.929505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:27.938908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 952/1902] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:27 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:27.943300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9be799-38e9-4473-9ac0-543d8f958536 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:27.962757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:27.966116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:27.966485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:27.966843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] PATH_INFO: `/v3/projects/fd5ef5fb13bd44aead2ea3b51fd06a17/users/d750f1a989264aa397c4b1327295e7c3/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:27.999871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21414569-b645-4fc8-91a0-d8063c50cc5f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.000342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21414569-b645-4fc8-91a0-d8063c50cc5f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.000844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21414569-b645-4fc8-91a0-d8063c50cc5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.024141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd5ef5fb13bd44aead2ea3b51fd06a17, user_id=d750f1a989264aa397c4b1327295e7c3, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:28.027975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:28.078118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 951/1903] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:27 2022] PUT /identity/v3/projects/fd5ef5fb13bd44aead2ea3b51fd06a17/users/d750f1a989264aa397c4b1327295e7c3/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:28.080746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:28.098214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-21414569-b645-4fc8-91a0-d8063c50cc5f None None] MFA Rules not processed for user `6930512abf8d4fdcadbf059ec4c4149f`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:28.101689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-21414569-b645-4fc8-91a0-d8063c50cc5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:28.323475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad2b118-07a8-41b2-888f-510f00505e1e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:28.329932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:28.331701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.332004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.332277 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.333206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:28.335894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:28.351971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 952/1904] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:28 2022] GET /identity/v3/roles => generated 1420 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:28.352952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:28.365004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:28.372913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d4f3c3-7174-478f-9cac-36cf130d72e2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:28.380764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:28.383526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.383964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.384361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] PATH_INFO: `/v3/projects/4e3b25808ff14c8b9aa65c8a2e3b2b19` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.385665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e3b25808ff14c8b9aa65c8a2e3b2b19)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:28.387839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:28.390565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 953/1905] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:27 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 392 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:28.391508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21414569-b645-4fc8-91a0-d8063c50cc5f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:28.437188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21414569-b645-4fc8-91a0-d8063c50cc5f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:28.450932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:28.453558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.454696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.456297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] PATH_INFO: `/v3/projects/fd5ef5fb13bd44aead2ea3b51fd06a17/users/d750f1a989264aa397c4b1327295e7c3/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.460893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd5ef5fb13bd44aead2ea3b51fd06a17, user_id=d750f1a989264aa397c4b1327295e7c3, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:28.467153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:28.475472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e3b25808ff14c8b9aa65c8a2e3b2b19', 'request_id': 'req-a4aa1b25-a0ca-4a46-b257-b287ac361b41'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:28.493871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e3b25808ff14c8b9aa65c8a2e3b2b19', 'request_id': 'req-a4aa1b25-a0ca-4a46-b257-b287ac361b41'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:28.526253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 954/1906] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:28 2022] PUT /identity/v3/projects/fd5ef5fb13bd44aead2ea3b51fd06a17/users/d750f1a989264aa397c4b1327295e7c3/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:28.528259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:28.531084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e3b25808ff14c8b9aa65c8a2e3b2b19', 'request_id': 'req-a4aa1b25-a0ca-4a46-b257-b287ac361b41'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:28.538203 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:28.554930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:28.568002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] The token cache is being invalidate because project 4e3b25808ff14c8b9aa65c8a2e3b2b19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:28.568653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:28.569872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4aa1b25-a0ca-4a46-b257-b287ac361b41 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:28.576560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 953/1907] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:28 2022] DELETE /identity/v3/projects/4e3b25808ff14c8b9aa65c8a2e3b2b19 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:28.578696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9febd151-0791-4fba-899f-847d6f74180b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.578696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9febd151-0791-4fba-899f-847d6f74180b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.579110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9febd151-0791-4fba-899f-847d6f74180b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.604251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:28.667841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9febd151-0791-4fba-899f-847d6f74180b None None] MFA Rules not processed for user `d750f1a989264aa397c4b1327295e7c3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:28.670957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9febd151-0791-4fba-899f-847d6f74180b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:28.718837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3e54b4-1a00-4195-97ea-c7b3b239cd7d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:28.738063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:28.738697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.738990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.739262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.751161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:28.882864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 954/1908] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:28 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:28.888864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3929736-2cf0-4bbd-b263-2433a29074b8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.889486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3929736-2cf0-4bbd-b263-2433a29074b8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.890537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3929736-2cf0-4bbd-b263-2433a29074b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.900582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:28.902645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:28.928248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 955/1909] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:28 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:28.930570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:28.934824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:28.947163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a3929736-2cf0-4bbd-b263-2433a29074b8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:28.953390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a3929736-2cf0-4bbd-b263-2433a29074b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:28.961620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4996ea-02fe-41b3-b909-409ca48ad1c8 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:28.966726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:28.968365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:28.968839 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:28.969423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:28.979955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:28.981733 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:29.009173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 956/1910] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:28 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:29.145735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:29.159881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a11c4d-e792-4dc0-afd3-a262593c0e3b service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:29.165472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:29.166321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:29.166762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:29.167171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:29.170600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:29.260045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 955/1911] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:29.263334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e191690-a32c-4c16-9c07-9a8cc08372c7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:29.263639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e191690-a32c-4c16-9c07-9a8cc08372c7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:29.263894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e191690-a32c-4c16-9c07-9a8cc08372c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:29.318365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0e191690-a32c-4c16-9c07-9a8cc08372c7 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:29.319523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0e191690-a32c-4c16-9c07-9a8cc08372c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:29.348521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 956/1912] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:29.352403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e191690-a32c-4c16-9c07-9a8cc08372c7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:29.359964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:29.361792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:29.372243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e191690-a32c-4c16-9c07-9a8cc08372c7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:29.376989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 957/1913] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:29 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:29.387716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:29.388440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:29.388792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:29.389156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:29.391066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:29.391432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:29.393222 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:29.443934 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 957/1914] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:29.448892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:29.474116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a6a571-9529-4caa-a5b7-ad01573ee882 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:29.487367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:29.489690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:29.490144 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:29.490518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:29.492271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:29.494783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:29.584991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 958/1915] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:29 2022] POST /identity/v3/projects => generated 373 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:29.589139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:29.606641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac72f2d-d81e-49b2-b829-623cafa29258 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:29.752727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:29.758364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:29.759637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:29.760221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:29.761650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:29.764014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:29.795530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:29.809617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:29.817808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 958/1916] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:29 2022] POST /identity/v3/users => generated 287 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:29.825668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab50a744-6ed5-4b4b-acf4-6e276d39efce admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:29.833890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:29.835071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:29.836456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:29.837935 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:29.840386 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:29.846608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:29.918135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:29.918457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 959/1917] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:29 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:29.934470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42152a3-1ba1-4931-84c9-5b69a3772ed5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:29.951443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923a78ad-037d-4dce-a0ef-bd3875f855e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:29.952165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923a78ad-037d-4dce-a0ef-bd3875f855e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:29.952486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923a78ad-037d-4dce-a0ef-bd3875f855e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:29.952959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923a78ad-037d-4dce-a0ef-bd3875f855e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:29.961263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923a78ad-037d-4dce-a0ef-bd3875f855e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:29.962823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923a78ad-037d-4dce-a0ef-bd3875f855e5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:30.002468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 959/1918] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:29 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:30.017177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:30.018290 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fd10d3-8e2d-40d3-9996-2b83c3af33ac admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:30.032506 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:30.033092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.033372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.033646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] PATH_INFO: `/v3/projects/dd16033759d041f4b61d98f0b6fa30cb/users/b39188b4257e47259c154a3ab9366a07/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.063427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:30.075151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd16033759d041f4b61d98f0b6fa30cb, user_id=b39188b4257e47259c154a3ab9366a07, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:30.078165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:30.095995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 960/1919] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:30 2022] PUT /identity/v3/projects/dd16033759d041f4b61d98f0b6fa30cb/users/b39188b4257e47259c154a3ab9366a07/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:30.110619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e39da5-e590-4d5f-95be-5c26f39a70ce None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.111040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e39da5-e590-4d5f-95be-5c26f39a70ce None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.111419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e39da5-e590-4d5f-95be-5c26f39a70ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.189621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-82e39da5-e590-4d5f-95be-5c26f39a70ce None None] MFA Rules not processed for user `b39188b4257e47259c154a3ab9366a07`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:30.191597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-82e39da5-e590-4d5f-95be-5c26f39a70ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:30.324393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 960/1920] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:30 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:30.342246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:30.344204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:30.440300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a425d52-b5a2-4c1b-8c6c-b095be5a3d3c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:30.444959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94912c30-9da3-4f73-966a-88ed2a6ef519 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:30.445638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94912c30-9da3-4f73-966a-88ed2a6ef519 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.445922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94912c30-9da3-4f73-966a-88ed2a6ef519 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.446341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94912c30-9da3-4f73-966a-88ed2a6ef519 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.459991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94912c30-9da3-4f73-966a-88ed2a6ef519 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:30.463814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94912c30-9da3-4f73-966a-88ed2a6ef519 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:30.476967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 961/1921] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:30.501027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e39da5-e590-4d5f-95be-5c26f39a70ce None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:30.613544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e39da5-e590-4d5f-95be-5c26f39a70ce None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:30.618703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:30.619356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.619632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.620102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] PATH_INFO: `/v3/users/b39188b4257e47259c154a3ab9366a07` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.621334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b39188b4257e47259c154a3ab9366a07)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:30.623402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:30.694234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b39188b4257e47259c154a3ab9366a07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:30.695497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b39188b4257e47259c154a3ab9366a07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:30.704694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b39188b4257e47259c154a3ab9366a07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:30.705509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b39188b4257e47259c154a3ab9366a07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:30.714873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b39188b4257e47259c154a3ab9366a07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:30.739417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:30.750957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 961/1922] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:30 2022] DELETE /identity/v3/users/b39188b4257e47259c154a3ab9366a07 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:30.755650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721008a-ba88-4d61-8493-3231a944dc9b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.756004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721008a-ba88-4d61-8493-3231a944dc9b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.756349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721008a-ba88-4d61-8493-3231a944dc9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.785267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6721008a-ba88-4d61-8493-3231a944dc9b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:30.787326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6721008a-ba88-4d61-8493-3231a944dc9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:30.824983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:30.828013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:30.890958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 962/1923] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:30.890958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6180c7-6913-472b-8c00-3dea5d91058c None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.890958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6180c7-6913-472b-8c00-3dea5d91058c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.890958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6180c7-6913-472b-8c00-3dea5d91058c None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.893091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 963/1924] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:18:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:18:30.899759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6180c7-6913-472b-8c00-3dea5d91058c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:30.903853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-db6180c7-6913-472b-8c00-3dea5d91058c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:30.940687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416418a-78dd-4e82-be9b-1756dfc6a69d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:30.948379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:30.949204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.949488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.949933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.957030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:30.979490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6180c7-6913-472b-8c00-3dea5d91058c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:30.984375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d536cdd-0247-4caa-ad29-6242579be986 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:30.985269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d536cdd-0247-4caa-ad29-6242579be986 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:30.985646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d536cdd-0247-4caa-ad29-6242579be986 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:30.985982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d536cdd-0247-4caa-ad29-6242579be986 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:30.996033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d536cdd-0247-4caa-ad29-6242579be986 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:30.997487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d536cdd-0247-4caa-ad29-6242579be986 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:31.009303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 964/1925] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:31.017112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:31.017439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:31.018489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:31.073089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:31.074883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:31.086639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:31.088205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:31.108358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 962/1926] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:30 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:31.121738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:31.227778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 965/1927] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:18:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:31.245938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:31.247569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:31.283918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02c7c01-6bb8-457d-b786-c46e77e39e4d service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:31.290916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f52efe5-8dc5-4dfe-8ea4-331b93b84560 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:31.291543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f52efe5-8dc5-4dfe-8ea4-331b93b84560 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:31.291984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f52efe5-8dc5-4dfe-8ea4-331b93b84560 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:31.292268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f52efe5-8dc5-4dfe-8ea4-331b93b84560 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:31.304137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f52efe5-8dc5-4dfe-8ea4-331b93b84560 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:31.305858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f52efe5-8dc5-4dfe-8ea4-331b93b84560 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:31.341572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 963/1928] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:32.327962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:32.330138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:32.362124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c523c94-66b8-4edb-b804-7d76e0a01f92 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:32.368550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:32.369573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:32.370386 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:32.370675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] PATH_INFO: `/v3/projects/dd16033759d041f4b61d98f0b6fa30cb` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:32.372019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd16033759d041f4b61d98f0b6fa30cb)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:32.374548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:32.501040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd16033759d041f4b61d98f0b6fa30cb', 'request_id': 'req-f7af4bae-351a-4f29-a5e6-6234af52ea13'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:32.523797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd16033759d041f4b61d98f0b6fa30cb', 'request_id': 'req-f7af4bae-351a-4f29-a5e6-6234af52ea13'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:32.549404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd16033759d041f4b61d98f0b6fa30cb', 'request_id': 'req-f7af4bae-351a-4f29-a5e6-6234af52ea13'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:32.567155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] The token cache is being invalidate because project dd16033759d041f4b61d98f0b6fa30cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:32.567517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:32.568365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f7af4bae-351a-4f29-a5e6-6234af52ea13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:32.570026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 966/1929] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:32 2022] DELETE /identity/v3/projects/dd16033759d041f4b61d98f0b6fa30cb => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:32.575208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e28914-984a-4001-9129-c6e0d4516c9c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:32.575761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e28914-984a-4001-9129-c6e0d4516c9c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:32.575761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e28914-984a-4001-9129-c6e0d4516c9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:32.618570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-31e28914-984a-4001-9129-c6e0d4516c9c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:32.624229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-31e28914-984a-4001-9129-c6e0d4516c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:32.702416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 964/1930] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:32.710287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eac263-461f-49dc-8f54-765db33be78f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:32.710840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eac263-461f-49dc-8f54-765db33be78f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:32.711180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eac263-461f-49dc-8f54-765db33be78f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:32.747672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e9eac263-461f-49dc-8f54-765db33be78f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:32.747672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e9eac263-461f-49dc-8f54-765db33be78f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:32.747672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 967/1931] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:32.749787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e28914-984a-4001-9129-c6e0d4516c9c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:32.759787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e28914-984a-4001-9129-c6e0d4516c9c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:32.764441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:32.765176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:32.765554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:32.765936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:32.766618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:32.766920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:32.768500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:32.780785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 965/1932] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:32.785256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eac263-461f-49dc-8f54-765db33be78f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:32.794781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eac263-461f-49dc-8f54-765db33be78f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:32.800166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:32.800786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:32.801106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:32.801449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:32.802012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:32.803923 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:32.834080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 968/1933] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:32 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:32.837482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:32.850942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4f56fd-dd02-4dff-90ec-9eb8895d12ff admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:32.877780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:32.924464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:32.926231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:32.936470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:32.937809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:32.938205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:32.938611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:32.939695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:32.944526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:32.987695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 966/1934] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:32 2022] POST /identity/v3/users => generated 292 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:32.990789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:33.014192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ca8776-e60d-4d8f-8e82-007d784e9401 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:33.025683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:33.027562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.028863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.029384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.033513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:33.041118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:33.081291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 967/1935] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:33.082670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:33.087947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff77573-b890-4a6d-a4ab-07ae91a71f33 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:33.114818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:33.116464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.117132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d065a596-19de-4b7a-ad96-b5dcd0ff0beb admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:33.118074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.118674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] PATH_INFO: `/v3/users/593547adf9f24e17ab65bc701c90b766` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.120283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=593547adf9f24e17ab65bc701c90b766)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:33.123697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:33.134804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:33.135719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.136000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.136681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.137676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:33.140072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:33.183927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 968/1936] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:33.191999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:33.196966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:33.207159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bb0fb7-8547-42d5-89b2-769ac5259dee admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:33.214625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823cc587-1257-4f7d-9504-94663466c7f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:33.215442 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823cc587-1257-4f7d-9504-94663466c7f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.215909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823cc587-1257-4f7d-9504-94663466c7f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.216247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823cc587-1257-4f7d-9504-94663466c7f1 admin admin] PATH_INFO: `/v3/projects/a8ec26cdee0a4057b0dd10ec86118efd/users/c21d410ad44948afa3a4be502a977da8/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.242115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '593547adf9f24e17ab65bc701c90b766'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:33.243935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823cc587-1257-4f7d-9504-94663466c7f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8ec26cdee0a4057b0dd10ec86118efd, user_id=c21d410ad44948afa3a4be502a977da8, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:33.249178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823cc587-1257-4f7d-9504-94663466c7f1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:33.276593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 969/1937] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:33 2022] PUT /identity/v3/projects/a8ec26cdee0a4057b0dd10ec86118efd/users/c21d410ad44948afa3a4be502a977da8/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:33.282814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '593547adf9f24e17ab65bc701c90b766'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:33.288167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.288629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.289089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.289663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '593547adf9f24e17ab65bc701c90b766'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:33.292578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '593547adf9f24e17ab65bc701c90b766'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:33.311657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6e428a3c-0e5a-4eaa-a264-2b8121bd29e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '593547adf9f24e17ab65bc701c90b766'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:33.347393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] MFA Rules not processed for user `c21d410ad44948afa3a4be502a977da8`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:33.351198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:33.353137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 969/1938] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:32 2022] DELETE /identity/v3/users/593547adf9f24e17ab65bc701c90b766 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:33.354571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.355206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.355637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.398940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:33.401109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:33.486123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 970/1939] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:33.500542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:33.510819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:33.553207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 970/1940] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:33.569412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:33.570860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:33.621814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520be731-39cc-4d86-bd4a-51e13e1df81f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:33.632013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:33.632713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.632962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.633237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.652308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:33.653811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:33.663985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:33.677805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8caa1f-e4bd-45b2-8192-eaaf5824fba1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:33.681849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:33.682203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 971/1941] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:33.682468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:33.682965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:33.683247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:33.699478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:33.701213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:33.716179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 971/1942] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:34.266027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:34.268066 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:34.286690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ebe1a3-6fee-4399-ac2b-8fe560aa27d9 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:34.293055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:34.294042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:34.294042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:34.294371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] PATH_INFO: `/v3/projects/defbf4f78edd4a1ea496d5188a17685a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:34.295515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] RBAC: Authorizing `identity:delete_project(project_id=defbf4f78edd4a1ea496d5188a17685a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:34.297617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:34.357165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'defbf4f78edd4a1ea496d5188a17685a', 'request_id': 'req-da57827c-8362-4081-b334-e6fff4ad8aee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.357612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'defbf4f78edd4a1ea496d5188a17685a', 'request_id': 'req-da57827c-8362-4081-b334-e6fff4ad8aee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.368513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'defbf4f78edd4a1ea496d5188a17685a', 'request_id': 'req-da57827c-8362-4081-b334-e6fff4ad8aee'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.382773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] The token cache is being invalidate because project defbf4f78edd4a1ea496d5188a17685a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:34.382970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.383726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.388971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 972/1943] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:34 2022] DELETE /identity/v3/projects/defbf4f78edd4a1ea496d5188a17685a => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:34.391636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:34.402635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fdf6ca-6307-4566-8113-9801431ce289 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:34.455876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:34.483011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:34.484391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:34.484807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:34.485397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] PATH_INFO: `/v3/users/d750f1a989264aa397c4b1327295e7c3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:34.486568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d750f1a989264aa397c4b1327295e7c3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:34.488505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:34.547688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd750f1a989264aa397c4b1327295e7c3'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.556832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd750f1a989264aa397c4b1327295e7c3'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.557739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd750f1a989264aa397c4b1327295e7c3'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.559448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd750f1a989264aa397c4b1327295e7c3'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.570712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8623980-5f10-4143-9504-796c8197286c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd750f1a989264aa397c4b1327295e7c3'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:34.623780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 972/1944] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:34 2022] DELETE /identity/v3/users/d750f1a989264aa397c4b1327295e7c3 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:35.146983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.148478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:35.165328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da57827c-8362-4081-b334-e6fff4ad8aee admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:35.282072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:35.282635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.282916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.283208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] PATH_INFO: `/v3/projects/fd5ef5fb13bd44aead2ea3b51fd06a17` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.284351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd5ef5fb13bd44aead2ea3b51fd06a17)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:35.286290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:35.334448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd5ef5fb13bd44aead2ea3b51fd06a17', 'request_id': 'req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:35.339868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd5ef5fb13bd44aead2ea3b51fd06a17', 'request_id': 'req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:35.350195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd5ef5fb13bd44aead2ea3b51fd06a17', 'request_id': 'req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:35.361963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] The token cache is being invalidate because project fd5ef5fb13bd44aead2ea3b51fd06a17 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:35.362247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:35.362914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-fbf30dec-f6e5-4bf2-b95e-120264d3faf5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:35.364445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 973/1945] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:35 2022] DELETE /identity/v3/projects/fd5ef5fb13bd44aead2ea3b51fd06a17 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:35.369211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de292f4-af6e-40da-840a-686b1cfa0b97 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.369503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de292f4-af6e-40da-840a-686b1cfa0b97 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.369809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de292f4-af6e-40da-840a-686b1cfa0b97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.389576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5de292f4-af6e-40da-840a-686b1cfa0b97 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:35.391267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5de292f4-af6e-40da-840a-686b1cfa0b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:35.448131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 973/1946] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:35.456599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b760406-3ccd-4c1c-ad20-810bf7f4f1ec None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.456881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b760406-3ccd-4c1c-ad20-810bf7f4f1ec None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.457177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b760406-3ccd-4c1c-ad20-810bf7f4f1ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.477988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-4b760406-3ccd-4c1c-ad20-810bf7f4f1ec None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:35.479165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4b760406-3ccd-4c1c-ad20-810bf7f4f1ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:35.487074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 974/1947] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:35.491363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de292f4-af6e-40da-840a-686b1cfa0b97 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.501732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de292f4-af6e-40da-840a-686b1cfa0b97 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:35.507529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:35.508148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.508431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.508684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.509370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:35.509674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:35.511273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:35.524043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 974/1948] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:35.528510 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b760406-3ccd-4c1c-ad20-810bf7f4f1ec None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.534197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:35.537880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b760406-3ccd-4c1c-ad20-810bf7f4f1ec None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:35.543890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:35.544644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.545109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.545437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.546869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:35.548743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:35.574621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 975/1949] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:35 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:35.578778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.588628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b58c10e-cc7b-4baf-b5ae-024c5f08204c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:35.635009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:35.648949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:35.649557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.649834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.650105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.650731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:35.652882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:35.676051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 975/1950] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:35 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:35.680240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.691033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495edd7a-c96a-4255-bfab-fce7ae72f155 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:35.695683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:35.696298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.696581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.696860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.697892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:35.699492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:35.710739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 976/1951] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:35.714840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.725214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0706c8-31c5-4e7c-bfe5-b7192cbb3553 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:35.730031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7356044b-21e9-46ad-a45f-cd178f7b1a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:35.730648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7356044b-21e9-46ad-a45f-cd178f7b1a71 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.730934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7356044b-21e9-46ad-a45f-cd178f7b1a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.731211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7356044b-21e9-46ad-a45f-cd178f7b1a71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.731869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7356044b-21e9-46ad-a45f-cd178f7b1a71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:35.733996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7356044b-21e9-46ad-a45f-cd178f7b1a71 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:35.744703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 976/1952] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:35.748848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.759581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9ee9f5-f56f-40f9-b931-45b6f1bf298c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:35.764283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:35.765008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.765291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.765586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] PATH_INFO: `/v3/projects/c4f435e89b704bb0a5fd4b6373f5e1a3/users/399c3f71fb5d495a9c3a7148f8357bde/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.780322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4f435e89b704bb0a5fd4b6373f5e1a3, user_id=399c3f71fb5d495a9c3a7148f8357bde, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:35.783004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:35.797367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 977/1953] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:35 2022] PUT /identity/v3/projects/c4f435e89b704bb0a5fd4b6373f5e1a3/users/399c3f71fb5d495a9c3a7148f8357bde/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:35.808280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:35.808698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:35.809618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:35.838660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] MFA Rules not processed for user `399c3f71fb5d495a9c3a7148f8357bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:35.839910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:35.916982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 977/1954] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:35 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:35.925937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:35.927682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:36.013748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a9c9de-36cc-4889-a08d-3e367052c283 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:36.017953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2eeb203-90b9-445a-ac4e-ba22e406f3a8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:36.018554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eeb203-90b9-445a-ac4e-ba22e406f3a8 service glance] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:36.018832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eeb203-90b9-445a-ac4e-ba22e406f3a8 service glance] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:36.019100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eeb203-90b9-445a-ac4e-ba22e406f3a8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:36.027491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2eeb203-90b9-445a-ac4e-ba22e406f3a8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:36.028887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2eeb203-90b9-445a-ac4e-ba22e406f3a8 service glance] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:36.037573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 978/1955] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:35 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:42.389261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:42.390939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:42.462471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a1944a-3183-4c43-9047-6e4fb7322f45 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:42.467972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:42.468515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:42.468853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:42.469157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] PATH_INFO: `/v3/users/7473a37c4e3947ca956a2ba74b0ed229` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:42.470272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7473a37c4e3947ca956a2ba74b0ed229)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:42.472200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:42.527737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7473a37c4e3947ca956a2ba74b0ed229'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:42.536136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7473a37c4e3947ca956a2ba74b0ed229'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:42.536872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7473a37c4e3947ca956a2ba74b0ed229'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:42.537529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7473a37c4e3947ca956a2ba74b0ed229'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:42.540102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:42.547789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7473a37c4e3947ca956a2ba74b0ed229'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:42.571436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 978/1956] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:42 2022] DELETE /identity/v3/users/7473a37c4e3947ca956a2ba74b0ed229 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:42.575876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:42.576385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:42.576715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:42.603742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:42.605018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:42.697819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 979/1957] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:42.709826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:42.715006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:42.796423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b63939d-d569-4c44-a89e-c7ba70c9cec1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:42.801788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:42.802997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:42.803524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:42.804114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:42.826325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:42.829749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:42.841447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 979/1958] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:43.295012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:43.297455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:43.316609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825ab83b-5ede-49be-8663-4361b16fe27f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:43.335847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:43.337777 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:43.338627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:43.339206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] PATH_INFO: `/v3/projects/9a980fc83cca48e0ab131de04cc994b8` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:43.342826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a980fc83cca48e0ab131de04cc994b8)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:43.344738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:43.380873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:43.409519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a980fc83cca48e0ab131de04cc994b8', 'request_id': 'req-a4528a79-8156-48cf-bf21-9576aee44ef8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.418053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a980fc83cca48e0ab131de04cc994b8', 'request_id': 'req-a4528a79-8156-48cf-bf21-9576aee44ef8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.434168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a980fc83cca48e0ab131de04cc994b8', 'request_id': 'req-a4528a79-8156-48cf-bf21-9576aee44ef8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.446707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] The token cache is being invalidate because project 9a980fc83cca48e0ab131de04cc994b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:43.446999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.447716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.449450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 980/1959] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:43 2022] DELETE /identity/v3/projects/9a980fc83cca48e0ab131de04cc994b8 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:43.454435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:43.465673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a404a-2129-43dc-b31e-3d948cf5d16f service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:43.620317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:43.621242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:43.621688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:43.622116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] PATH_INFO: `/v3/users/ab157d08158349c7b292955165982618` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:43.623989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab157d08158349c7b292955165982618)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:43.626082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:43.687825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab157d08158349c7b292955165982618'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.695227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab157d08158349c7b292955165982618'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.695970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab157d08158349c7b292955165982618'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.696578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab157d08158349c7b292955165982618'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.705204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d16e7b42-8af8-41b2-85f6-2b1e52ff5f2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab157d08158349c7b292955165982618'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:43.719625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:43.725242 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 980/1960] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:43 2022] DELETE /identity/v3/users/ab157d08158349c7b292955165982618 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:44.013540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.015189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:44.032338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4528a79-8156-48cf-bf21-9576aee44ef8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.102537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.103255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.103623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.103983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] PATH_INFO: `/v3/projects/065925f0023e436d8e045acd3617d12d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.105484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] RBAC: Authorizing `identity:delete_project(project_id=065925f0023e436d8e045acd3617d12d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:44.107631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:44.162646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '065925f0023e436d8e045acd3617d12d', 'request_id': 'req-85819756-d351-448e-a6ff-6848c4e7e14a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.169133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '065925f0023e436d8e045acd3617d12d', 'request_id': 'req-85819756-d351-448e-a6ff-6848c4e7e14a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.180153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '065925f0023e436d8e045acd3617d12d', 'request_id': 'req-85819756-d351-448e-a6ff-6848c4e7e14a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.205398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] The token cache is being invalidate because project 065925f0023e436d8e045acd3617d12d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:44.205690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.206352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85819756-d351-448e-a6ff-6848c4e7e14a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.207953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 981/1961] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:44 2022] DELETE /identity/v3/projects/065925f0023e436d8e045acd3617d12d => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:44.212959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d6331-1258-4632-92d0-d8cc2ede9048 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.213251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d6331-1258-4632-92d0-d8cc2ede9048 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.213545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012d6331-1258-4632-92d0-d8cc2ede9048 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.236145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-012d6331-1258-4632-92d0-d8cc2ede9048 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:44.237654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-012d6331-1258-4632-92d0-d8cc2ede9048 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:44.303830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 981/1962] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:44.312146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f2f17-04d4-4f04-b0ae-6c5dee587f16 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.312446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f2f17-04d4-4f04-b0ae-6c5dee587f16 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.312729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f2f17-04d4-4f04-b0ae-6c5dee587f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.333172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9c7f2f17-04d4-4f04-b0ae-6c5dee587f16 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:44.334367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9c7f2f17-04d4-4f04-b0ae-6c5dee587f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:44.345190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 982/1963] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:44.349668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012d6331-1258-4632-92d0-d8cc2ede9048 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.360154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012d6331-1258-4632-92d0-d8cc2ede9048 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.375959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.376606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.376890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.377200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.377860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:44.378173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:44.379823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:44.395773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 982/1964] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:44.396959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f2f17-04d4-4f04-b0ae-6c5dee587f16 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.428163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f2f17-04d4-4f04-b0ae-6c5dee587f16 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.434294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.434940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.435512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.435791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.436591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:44.438830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:44.486646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 983/1965] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:44 2022] POST /identity/v3/projects => generated 385 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:44.487319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.508477 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23e6955-dfd7-41f4-82ac-fb589a34028a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.513265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.516155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:44.549553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:44.597197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.598211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.598503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.598780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.599952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:44.601957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:44.612978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280857bb-7f00-4b2c-8e36-c08300f85779 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.620967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.621876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.622251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.622622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] PATH_INFO: `/v3/users/c21d410ad44948afa3a4be502a977da8` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.624054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c21d410ad44948afa3a4be502a977da8)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:44.626398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:44.641624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 983/1966] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:44 2022] POST /identity/v3/users => generated 293 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:44.644016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.667180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2463fb2-0168-4a71-8ea3-e5c4f16b0928 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.673982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.674749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.675745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.675745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.676594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:44.679013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:44.693901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 984/1967] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:44 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:44.698438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.711420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6890fda3-c38b-4b52-a761-7c4928efa1ef admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.713586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c21d410ad44948afa3a4be502a977da8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.714464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c21d410ad44948afa3a4be502a977da8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.719664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.720392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.720884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.721148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.722394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:44.723890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:44.724711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c21d410ad44948afa3a4be502a977da8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.725838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c21d410ad44948afa3a4be502a977da8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.739406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-33249d93-3a12-48c9-98f0-5097722b1b4f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c21d410ad44948afa3a4be502a977da8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:44.741541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 985/1968] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:44 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:18:44.745240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:44.748672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:44.760482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 984/1969] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:44 2022] DELETE /identity/v3/users/c21d410ad44948afa3a4be502a977da8 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:44.765629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.765985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.769688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.774032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c88805e-5d92-4797-9083-a8ff70a9b8f5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:44.792670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:44.793941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:44.925239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:44.962336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a118cdbf-0ae9-4983-93ed-dd1190fb4a4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:44.963251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a118cdbf-0ae9-4983-93ed-dd1190fb4a4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:44.963637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a118cdbf-0ae9-4983-93ed-dd1190fb4a4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:44.964107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a118cdbf-0ae9-4983-93ed-dd1190fb4a4f admin admin] PATH_INFO: `/v3/projects/6b458fdf8a0a49519fd7113e7e04c59e/users/a1c9c2f1d46d43bba72d6c372cb511ba/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:44.970237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 985/1970] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:45.020451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:45.022762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:45.028354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a118cdbf-0ae9-4983-93ed-dd1190fb4a4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b458fdf8a0a49519fd7113e7e04c59e, user_id=a1c9c2f1d46d43bba72d6c372cb511ba, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:45.032217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a118cdbf-0ae9-4983-93ed-dd1190fb4a4f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:45.071492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 986/1971] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:44 2022] PUT /identity/v3/projects/6b458fdf8a0a49519fd7113e7e04c59e/users/a1c9c2f1d46d43bba72d6c372cb511ba/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:45.082530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6a44f1-4dd9-46d4-b917-0ee2ec4cd808 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:45.082832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6a44f1-4dd9-46d4-b917-0ee2ec4cd808 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:45.083092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6a44f1-4dd9-46d4-b917-0ee2ec4cd808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:45.169266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7e6a44f1-4dd9-46d4-b917-0ee2ec4cd808 None None] MFA Rules not processed for user `a1c9c2f1d46d43bba72d6c372cb511ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:45.174294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7e6a44f1-4dd9-46d4-b917-0ee2ec4cd808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:45.208069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:45.282986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138c89f-a5ba-42d4-a41e-f00f0beefa17 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:45.378671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 987/1972] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:45.379672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6a44f1-4dd9-46d4-b917-0ee2ec4cd808 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:45.470613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:45.471845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:45.472295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:45.474172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:45.533617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6a44f1-4dd9-46d4-b917-0ee2ec4cd808 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:45.538541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:45.539220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:45.539508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:45.539783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:45.540427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:45.542592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:45.599919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 988/1973] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:45 2022] POST /identity/v3/projects => generated 387 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:45.600871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:45.602537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:45.604413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:45.615083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 986/1974] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 595 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:45.673370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:45.691100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9045f9e-8021-4fc8-8911-eee24a71750b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:45.696165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:45.696901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:45.697267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:45.697606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:45.698327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:45.700304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:45.724213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 989/1975] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:45 2022] POST /identity/v3/users => generated 300 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:45.728494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:45.745262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55446b1d-3dd6-407f-b386-8111f5f38995 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:45.752267 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:45.752887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:45.753197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:45.753467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:45.754337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:45.756244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:45.767079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 987/1976] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:45.771442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:45.786812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ba47a4-6130-4602-b16a-26a23f878096 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:45.792609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:45.793510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:45.793798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:45.794127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] PATH_INFO: `/v3/projects/6a391a9f572c449b8b37af9a1bbf8e28/users/513813b7d5e64865b3d109beb7748672/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:45.817135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a391a9f572c449b8b37af9a1bbf8e28, user_id=513813b7d5e64865b3d109beb7748672, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:45.837962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:45.867395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 990/1977] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:45 2022] PUT /identity/v3/projects/6a391a9f572c449b8b37af9a1bbf8e28/users/513813b7d5e64865b3d109beb7748672/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:45.871575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:45.885556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f534e8fb-cac5-4b05-85a1-10e040f9b6f3 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:46.012221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258c4800-09ed-41bd-9800-5dcf1811d1b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:46.012847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258c4800-09ed-41bd-9800-5dcf1811d1b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.013164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258c4800-09ed-41bd-9800-5dcf1811d1b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.013517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258c4800-09ed-41bd-9800-5dcf1811d1b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.014720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258c4800-09ed-41bd-9800-5dcf1811d1b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:46.016983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258c4800-09ed-41bd-9800-5dcf1811d1b7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:46.031070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 988/1978] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:45 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:18:46.032081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:46.046003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f62ca3-b9d2-438c-b4ee-1fbd6ff1cca5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:46.058118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:46.059351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.060219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.060517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] PATH_INFO: `/v3/projects/6a391a9f572c449b8b37af9a1bbf8e28/users/513813b7d5e64865b3d109beb7748672/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.065767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a391a9f572c449b8b37af9a1bbf8e28, user_id=513813b7d5e64865b3d109beb7748672, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:46.070218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:46.095850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 991/1979] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:46 2022] PUT /identity/v3/projects/6a391a9f572c449b8b37af9a1bbf8e28/users/513813b7d5e64865b3d109beb7748672/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:46.098337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.098718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.099087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.129545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] MFA Rules not processed for user `513813b7d5e64865b3d109beb7748672`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:46.130823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:46.146404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:46.252139 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 989/1980] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:46 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:46.267273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:46.269525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:46.362141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c6dcd8-e1f0-4cdb-90b8-a0c834c0d2e8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:46.366356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:46.367069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.367371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.367627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.390671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:46.392518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:46.408703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 992/1981] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:46 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:46.429546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:46.431688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:46.596201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2ace37-b74f-404f-9679-d8884f139717 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:46.607595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:46.616419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:46.617256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.617548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.617808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] PATH_INFO: `/v3/projects/a8ec26cdee0a4057b0dd10ec86118efd` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.618990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f2483a-f35b-45b7-807f-e190980034af service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:46.619769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8ec26cdee0a4057b0dd10ec86118efd)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:46.621716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:46.624443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:46.625299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.626086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.626086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] PATH_INFO: `/v3/users/a1c9c2f1d46d43bba72d6c372cb511ba` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.627103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1c9c2f1d46d43bba72d6c372cb511ba)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:46.629056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:46.723287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1c9c2f1d46d43bba72d6c372cb511ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.736096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1c9c2f1d46d43bba72d6c372cb511ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.737337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1c9c2f1d46d43bba72d6c372cb511ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.738167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1c9c2f1d46d43bba72d6c372cb511ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.743834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8ec26cdee0a4057b0dd10ec86118efd', 'request_id': 'req-f68373c6-bf92-41d1-83f7-e37053a78642'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.754493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6790952e-b543-47f0-b389-e2353ed12dad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1c9c2f1d46d43bba72d6c372cb511ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.784302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8ec26cdee0a4057b0dd10ec86118efd', 'request_id': 'req-f68373c6-bf92-41d1-83f7-e37053a78642'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.787721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 993/1982] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:46 2022] DELETE /identity/v3/users/a1c9c2f1d46d43bba72d6c372cb511ba => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:46.796386 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.797384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.798313 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.806411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8ec26cdee0a4057b0dd10ec86118efd', 'request_id': 'req-f68373c6-bf92-41d1-83f7-e37053a78642'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.851912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] The token cache is being invalidate because project a8ec26cdee0a4057b0dd10ec86118efd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:46.855388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:46.856208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.856694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f68373c6-bf92-41d1-83f7-e37053a78642 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:46.857080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:46.865625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 990/1983] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:46 2022] DELETE /identity/v3/projects/a8ec26cdee0a4057b0dd10ec86118efd => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:46.871015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84920b4-ebee-4d6e-adc9-c9cde2a78443 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:46.871421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84920b4-ebee-4d6e-adc9-c9cde2a78443 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:46.871768 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84920b4-ebee-4d6e-adc9-c9cde2a78443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:46.883823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=84,keystone:INSERT=6,keystone:DELETE=10 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:46.923329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e84920b4-ebee-4d6e-adc9-c9cde2a78443 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:46.925828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e84920b4-ebee-4d6e-adc9-c9cde2a78443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:46.986095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 994/1984] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:46.999610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.001755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:47.018613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:47.051448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 991/1985] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:47.059169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845cdbc1-651f-4633-8f8a-bbf71922bc9f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.059571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845cdbc1-651f-4633-8f8a-bbf71922bc9f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.059913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845cdbc1-651f-4633-8f8a-bbf71922bc9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.089965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-845cdbc1-651f-4633-8f8a-bbf71922bc9f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:47.091746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-845cdbc1-651f-4633-8f8a-bbf71922bc9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:47.104207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 992/1986] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:47.109369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845cdbc1-651f-4633-8f8a-bbf71922bc9f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.129183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845cdbc1-651f-4633-8f8a-bbf71922bc9f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.130219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd481c87-d723-4746-90b0-c4050ad5457c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.134838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.136017 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.136694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.137212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.137678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.138426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.138826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.140039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.141244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:47.142073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.144753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.159774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.161698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.165543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 993/1987] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:18:47.167283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.180009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 995/1988] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:47.191241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b863f9e-da6e-4e2c-82b2-33163700f11c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.197412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c6f370-379e-48eb-85b4-9a68c8f6a2b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.198356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6f370-379e-48eb-85b4-9a68c8f6a2b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.198781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6f370-379e-48eb-85b4-9a68c8f6a2b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.199328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6f370-379e-48eb-85b4-9a68c8f6a2b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.200180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c6f370-379e-48eb-85b4-9a68c8f6a2b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.203206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c6f370-379e-48eb-85b4-9a68c8f6a2b0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.298620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.310823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 994/1989] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:47 2022] POST /identity/v3/projects => generated 367 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:47.322521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed247f4-93e0-4e3f-9bc8-1c118311efa3 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.453448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb3c222-1544-423d-812f-94592a55cc7e None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.453916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb3c222-1544-423d-812f-94592a55cc7e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.454201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb3c222-1544-423d-812f-94592a55cc7e None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.456868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 995/1990] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:18:47 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:18:47.483919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.484259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.484531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.544613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.545476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.545767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.546037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.546603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:47.546750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.548024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:47.549160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.608669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 996/1991] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:47 2022] POST /identity/v3/users => generated 284 bytes in 311 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:47.609665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.645923 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0898c06c-d01f-430f-86b6-5e5d513d370b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.662502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.670742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.672131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.672429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.674477 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.679051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.701164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 997/1992] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:47.703419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.722735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 996/1993] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:18:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:47.730125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e893cf3a-dd68-48be-ba29-2ca431074c69 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.746653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.747504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.747911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.751811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.752639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.758086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.780075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.783052 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:47.797237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 998/1994] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:47.799253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:47.819427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e2003b-8671-4b9e-89df-1d5a8e3b9d12 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.829379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8398af5-c231-44cf-8d76-1d526bfe0126 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:47.833113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676ad5df-4a46-4b9b-917e-c31d4c511c86 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.834769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676ad5df-4a46-4b9b-917e-c31d4c511c86 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.835645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676ad5df-4a46-4b9b-917e-c31d4c511c86 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.843306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676ad5df-4a46-4b9b-917e-c31d4c511c86 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.845649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:47.847336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.850818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.851259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] PATH_INFO: `/v3/projects/72082f52f6244af1a940068cf4c2ac0c/users/1344a7ba15f844998e2526391f283a7c/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:47.872247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676ad5df-4a46-4b9b-917e-c31d4c511c86 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.874638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676ad5df-4a46-4b9b-917e-c31d4c511c86 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.894774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 997/1995] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:47.897171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72082f52f6244af1a940068cf4c2ac0c, user_id=1344a7ba15f844998e2526391f283a7c, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:47.900447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:47.933457 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 999/1996] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:47 2022] PUT /identity/v3/projects/72082f52f6244af1a940068cf4c2ac0c/users/1344a7ba15f844998e2526391f283a7c/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:47.947007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:47.947295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:47.947572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:48.035308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:48.037470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:48.046214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:48.049851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] MFA Rules not processed for user `1344a7ba15f844998e2526391f283a7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:48.051470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:48.156709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 998/1997] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:47 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:48.167763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:48.170500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:48.181554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3404ee-1548-4ef3-b4e0-8d6487d3d8b1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:48.201230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:48.201230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:48.201230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:48.201230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] PATH_INFO: `/v3/projects/6b458fdf8a0a49519fd7113e7e04c59e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:48.202264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b458fdf8a0a49519fd7113e7e04c59e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:48.202264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:48.202761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:48.271308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b458fdf8a0a49519fd7113e7e04c59e', 'request_id': 'req-0555fb70-1720-49f8-ab82-3cabc17dc311'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.271907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fd71a9-c267-4ff4-8e4c-e99f15bb2312 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:48.286323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b458fdf8a0a49519fd7113e7e04c59e', 'request_id': 'req-0555fb70-1720-49f8-ab82-3cabc17dc311'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.302521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b458fdf8a0a49519fd7113e7e04c59e', 'request_id': 'req-0555fb70-1720-49f8-ab82-3cabc17dc311'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.334708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] The token cache is being invalidate because project 6b458fdf8a0a49519fd7113e7e04c59e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:48.335484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.338240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.347583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1000/1998] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:48 2022] DELETE /identity/v3/projects/6b458fdf8a0a49519fd7113e7e04c59e => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:48.349170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:48.351139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:48.355751 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:48.358601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:48.358968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:48.359255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:48.363136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:48.461030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0555fb70-1720-49f8-ab82-3cabc17dc311 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:48.468115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:48.468773 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:48.469219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:48.469502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] PATH_INFO: `/v3/users/513813b7d5e64865b3d109beb7748672` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:48.470807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] RBAC: Authorizing `identity:delete_user(user_id=513813b7d5e64865b3d109beb7748672)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:48.471905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:48.473167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:48.485803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:48.497165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 999/1999] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:48 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:48.557694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:48.578800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '513813b7d5e64865b3d109beb7748672'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.594051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '513813b7d5e64865b3d109beb7748672'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.595599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '513813b7d5e64865b3d109beb7748672'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.596470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '513813b7d5e64865b3d109beb7748672'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.609812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-5be3b91a-1336-44a8-840a-293271782369 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '513813b7d5e64865b3d109beb7748672'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:48.652473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1001/2000] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:48 2022] DELETE /identity/v3/users/513813b7d5e64865b3d109beb7748672 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:49.341662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:49.360778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ed45d-5897-4dc4-a2e6-b06d0ff41c7b service glance] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:49.507974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:49.508852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:49.509353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:49.509692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] PATH_INFO: `/v3/projects/6a391a9f572c449b8b37af9a1bbf8e28` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:49.514894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] RBAC: Authorizing `identity:delete_project(project_id=6a391a9f572c449b8b37af9a1bbf8e28)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:49.518272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:49.551142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:49.577651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6a391a9f572c449b8b37af9a1bbf8e28', 'request_id': 'req-7c36f776-a68a-4811-ba0e-7ddea430062c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:49.583320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6a391a9f572c449b8b37af9a1bbf8e28', 'request_id': 'req-7c36f776-a68a-4811-ba0e-7ddea430062c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:49.591171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6a391a9f572c449b8b37af9a1bbf8e28', 'request_id': 'req-7c36f776-a68a-4811-ba0e-7ddea430062c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:49.602475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] The token cache is being invalidate because project 6a391a9f572c449b8b37af9a1bbf8e28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:49.602764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:49.603400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7c36f776-a68a-4811-ba0e-7ddea430062c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:49.604880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1000/2001] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:49 2022] DELETE /identity/v3/projects/6a391a9f572c449b8b37af9a1bbf8e28 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:49.610112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc58d1-4969-447b-a1f1-8210509ce50d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:49.610584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc58d1-4969-447b-a1f1-8210509ce50d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:49.610694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc58d1-4969-447b-a1f1-8210509ce50d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:49.630191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-54bc58d1-4969-447b-a1f1-8210509ce50d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:49.631404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-54bc58d1-4969-447b-a1f1-8210509ce50d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:49.694168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1002/2002] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:49.702991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e82654-9cbd-4e9f-9b5a-f91421b8ab57 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:49.703419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e82654-9cbd-4e9f-9b5a-f91421b8ab57 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:49.703820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e82654-9cbd-4e9f-9b5a-f91421b8ab57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:49.724547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-06e82654-9cbd-4e9f-9b5a-f91421b8ab57 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:49.725775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-06e82654-9cbd-4e9f-9b5a-f91421b8ab57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:49.734869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1001/2003] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:49.739122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc58d1-4969-447b-a1f1-8210509ce50d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:49.749662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc58d1-4969-447b-a1f1-8210509ce50d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:49.754713 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:49.755347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:49.755660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:49.755958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:49.756745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:49.757114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:49.758953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:49.772234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1003/2004] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:49.776819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e82654-9cbd-4e9f-9b5a-f91421b8ab57 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:49.786245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e82654-9cbd-4e9f-9b5a-f91421b8ab57 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:49.791029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:49.791664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:49.791947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:49.792221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:49.792884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:49.794908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:49.822460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1002/2005] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:49 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:49.827249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:49.837054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3446e53-d464-44e1-817c-da687d82a066 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:49.911078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:49.911731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:49.912024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:49.912309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:49.913028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:49.915052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:49.940497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1004/2006] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:49 2022] POST /identity/v3/users => generated 294 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:49.945543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:49.956088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc177d86-da63-4b5e-b686-1b58ebf405f4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:49.961359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:49.961969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:49.962246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:49.962515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:49.963787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:49.966085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:49.984069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1003/2007] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:49 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:49.988364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:50.007391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350c00e9-b0a6-4803-a49a-86c72b0fe819 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:50.012712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40af6f71-918c-4ffb-9a46-01ba15fd987d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:50.013946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40af6f71-918c-4ffb-9a46-01ba15fd987d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.013946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40af6f71-918c-4ffb-9a46-01ba15fd987d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.014251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40af6f71-918c-4ffb-9a46-01ba15fd987d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.015711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40af6f71-918c-4ffb-9a46-01ba15fd987d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:50.017215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40af6f71-918c-4ffb-9a46-01ba15fd987d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:50.040257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:50.041346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1005/2008] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:49 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:50.055108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf08518-508b-4919-9ee2-294bd5cb6cb9 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:50.060390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:50.062764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.063050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.063327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] PATH_INFO: `/v3/projects/736dec385e1b40a5a780fed18a76bae2/users/7b0b575aca894f6ba3f7dc9b3ae5b87d/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.080956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=736dec385e1b40a5a780fed18a76bae2, user_id=7b0b575aca894f6ba3f7dc9b3ae5b87d, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:50.084976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:50.109220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1004/2009] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:50 2022] PUT /identity/v3/projects/736dec385e1b40a5a780fed18a76bae2/users/7b0b575aca894f6ba3f7dc9b3ae5b87d/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:50.116207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c03f3e-3573-4782-aada-11590db0ca6f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.116501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c03f3e-3573-4782-aada-11590db0ca6f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.116861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c03f3e-3573-4782-aada-11590db0ca6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.156127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-73c03f3e-3573-4782-aada-11590db0ca6f None None] MFA Rules not processed for user `7b0b575aca894f6ba3f7dc9b3ae5b87d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:50.157541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73c03f3e-3573-4782-aada-11590db0ca6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:50.167283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:50.293061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1006/2010] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:50 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:50.298545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:50.399129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69499339-4ac8-4ede-903e-ee882cad90f0 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:50.407832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:50.408637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.409108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.409556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.410387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:50.412523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:50.460067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1005/2011] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:50 2022] POST /identity/v3/projects => generated 385 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:50.465468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c03f3e-3573-4782-aada-11590db0ca6f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:50.480504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c03f3e-3573-4782-aada-11590db0ca6f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:50.564306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:50.565234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.565507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.565783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.566612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:50.568572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:50.596193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1007/2012] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:50 2022] POST /identity/v3/users => generated 293 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:50.599821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:50.612435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162f636f-ba40-4ae6-9fb1-c7d98cb0dd32 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:50.617365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:50.617985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.618266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.618537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.619188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:50.620787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:50.637874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1006/2013] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:50.639586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:50.652807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2772d9c-770e-4d5e-aacd-271095cd4dd3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:50.659602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a935d1-0ba2-41e4-8b43-ae9784864ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:50.660103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a935d1-0ba2-41e4-8b43-ae9784864ab0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.660483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a935d1-0ba2-41e4-8b43-ae9784864ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.660887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a935d1-0ba2-41e4-8b43-ae9784864ab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.661712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a935d1-0ba2-41e4-8b43-ae9784864ab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:50.663401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a935d1-0ba2-41e4-8b43-ae9784864ab0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:50.677532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1008/2014] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:50.678880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:50.691608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1caafd-fda0-4ad1-b9fe-47b3344790df admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:50.702497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:50.703623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.704612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.705105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] PATH_INFO: `/v3/projects/061e5ad9426c4a9cb1eeef269dfc6f42/users/b6c0713f0417492ca2ab6fcc1312fa64/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.728204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=061e5ad9426c4a9cb1eeef269dfc6f42, user_id=b6c0713f0417492ca2ab6fcc1312fa64, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:50.731747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:50.739126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=7,keystone:SELECT=88,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:50.761426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1007/2015] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:50 2022] PUT /identity/v3/projects/061e5ad9426c4a9cb1eeef269dfc6f42/users/b6c0713f0417492ca2ab6fcc1312fa64/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:50.767359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8d7d0-048f-460c-a5ca-9eb8e947d2f0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:50.767990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8d7d0-048f-460c-a5ca-9eb8e947d2f0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:50.768274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8d7d0-048f-460c-a5ca-9eb8e947d2f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:50.812856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e0a8d7d0-048f-460c-a5ca-9eb8e947d2f0 None None] MFA Rules not processed for user `b6c0713f0417492ca2ab6fcc1312fa64`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:50.814304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e0a8d7d0-048f-460c-a5ca-9eb8e947d2f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:50.905611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1009/2016] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:50.906464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.017587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4960c21e-4b08-4b65-9c8f-5c5e77c48b83 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.024502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.025514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.025805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.026363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.027277 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.029466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.062590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1008/2017] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:50 2022] POST /identity/v3/projects => generated 385 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:51.067335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a8d7d0-048f-460c-a5ca-9eb8e947d2f0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.082189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a8d7d0-048f-460c-a5ca-9eb8e947d2f0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.112628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:51.174923 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.176126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.176456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.176987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.178142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.181813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.210261 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1010/2018] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:51 2022] POST /identity/v3/users => generated 299 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:51.214634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.230520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4844bf-031b-4bf2-b146-0df42095f383 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.237422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68f3c8f-36c4-4c8f-a2d2-d6c8522193be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.238044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f3c8f-36c4-4c8f-a2d2-d6c8522193be admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.238369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f3c8f-36c4-4c8f-a2d2-d6c8522193be admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.238558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68f3c8f-36c4-4c8f-a2d2-d6c8522193be admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.239213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68f3c8f-36c4-4c8f-a2d2-d6c8522193be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.240824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68f3c8f-36c4-4c8f-a2d2-d6c8522193be admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.255030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1009/2019] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:51 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:51.259420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.273811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d9e099-1307-408f-9b7a-932d606b465b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.284665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bf2a3c-2ab4-4084-b1d5-e77b3af7da58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.285542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf2a3c-2ab4-4084-b1d5-e77b3af7da58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.285824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf2a3c-2ab4-4084-b1d5-e77b3af7da58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.286101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bf2a3c-2ab4-4084-b1d5-e77b3af7da58 admin admin] PATH_INFO: `/v3/projects/e331719e89fc43838931b41bbf397864/users/01fdb5f4ff854e13bf802348b54b73f4/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.314604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bf2a3c-2ab4-4084-b1d5-e77b3af7da58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e331719e89fc43838931b41bbf397864, user_id=01fdb5f4ff854e13bf802348b54b73f4, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.315657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bf2a3c-2ab4-4084-b1d5-e77b3af7da58 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.328266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940905c8-a9db-4227-8ad4-24fd1c7b87f9 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.328826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940905c8-a9db-4227-8ad4-24fd1c7b87f9 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.329280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940905c8-a9db-4227-8ad4-24fd1c7b87f9 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.331854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1010/2020] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:18:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:18:51.337623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1011/2021] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:51 2022] PUT /identity/v3/projects/e331719e89fc43838931b41bbf397864/users/01fdb5f4ff854e13bf802348b54b73f4/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:51.342252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940905c8-a9db-4227-8ad4-24fd1c7b87f9 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.361415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37823270-dd9f-4ab5-9d28-bf58ea887173 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.361834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37823270-dd9f-4ab5-9d28-bf58ea887173 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.362178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37823270-dd9f-4ab5-9d28-bf58ea887173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.389933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-37823270-dd9f-4ab5-9d28-bf58ea887173 None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:51.391371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-37823270-dd9f-4ab5-9d28-bf58ea887173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:51.477230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940905c8-a9db-4227-8ad4-24fd1c7b87f9 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.484433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.484689 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.492421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.492421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.492421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.509488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.536014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1011/2022] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:51 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:18:51.537616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.557560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c3202f-e80c-48bd-85d9-7ac2f386b1d4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.558266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1012/2023] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:18:51 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:51.575756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49447f75-2b00-4e89-b900-66ac96e04f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.577913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49447f75-2b00-4e89-b900-66ac96e04f76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.578228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49447f75-2b00-4e89-b900-66ac96e04f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.578514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49447f75-2b00-4e89-b900-66ac96e04f76 admin admin] PATH_INFO: `/v3/projects/e331719e89fc43838931b41bbf397864/users/01fdb5f4ff854e13bf802348b54b73f4/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.581567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49447f75-2b00-4e89-b900-66ac96e04f76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e331719e89fc43838931b41bbf397864, user_id=01fdb5f4ff854e13bf802348b54b73f4, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.584173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad093253-5506-44b6-9c8f-5a123df59ff7 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.584468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad093253-5506-44b6-9c8f-5a123df59ff7 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.584776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad093253-5506-44b6-9c8f-5a123df59ff7 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.587683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49447f75-2b00-4e89-b900-66ac96e04f76 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.588444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1013/2024] 10.222.0.26 () {54 vars in 961 bytes} [Sat Oct 1 04:18:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:18:51.595767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6924d57a-6d3c-4764-80c6-55cb3cd2c1a8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.596065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6924d57a-6d3c-4764-80c6-55cb3cd2c1a8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.596329 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6924d57a-6d3c-4764-80c6-55cb3cd2c1a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.611289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1012/2025] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:51 2022] PUT /identity/v3/projects/e331719e89fc43838931b41bbf397864/users/01fdb5f4ff854e13bf802348b54b73f4/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:51.621389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.621750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.622278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.635763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6924d57a-6d3c-4764-80c6-55cb3cd2c1a8 None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:51.637047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6924d57a-6d3c-4764-80c6-55cb3cd2c1a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:51.688631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] MFA Rules not processed for user `01fdb5f4ff854e13bf802348b54b73f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:51.689976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:51.744344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:51.787113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1014/2026] 10.222.0.26 () {58 vars in 1058 bytes} [Sat Oct 1 04:18:51 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:51.789859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1013/2027] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:51 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:51.800341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6924d57a-6d3c-4764-80c6-55cb3cd2c1a8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.805496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:51.807338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:51.818329 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6924d57a-6d3c-4764-80c6-55cb3cd2c1a8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.826942 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799635a-5e1b-4ae8-91cd-c5b6887ed710 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.828102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799635a-5e1b-4ae8-91cd-c5b6887ed710 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.828544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799635a-5e1b-4ae8-91cd-c5b6887ed710 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.829113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799635a-5e1b-4ae8-91cd-c5b6887ed710 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.840399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799635a-5e1b-4ae8-91cd-c5b6887ed710 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.842989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799635a-5e1b-4ae8-91cd-c5b6887ed710 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.855739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1015/2028] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:51.900439 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c82990-7333-47b6-a6f3-e0ebdd82f8d7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:51.905641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645db19-91a5-460b-a7f1-a5b0078638b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:51.906820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645db19-91a5-460b-a7f1-a5b0078638b7 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:51.907176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645db19-91a5-460b-a7f1-a5b0078638b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:51.907525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645db19-91a5-460b-a7f1-a5b0078638b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:51.917637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5645db19-91a5-460b-a7f1-a5b0078638b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:51.921389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5645db19-91a5-460b-a7f1-a5b0078638b7 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:51.930731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1014/2029] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:51 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:18:52.146832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ebe73-2c12-4dd7-824d-cd0ef5863233 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:52.147048 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ebe73-2c12-4dd7-824d-cd0ef5863233 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:52.147324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ebe73-2c12-4dd7-824d-cd0ef5863233 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:52.148818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1016/2030] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:18:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:18:52.167217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:52.167580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:52.167930 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:52.196060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] MFA Rules not processed for user `4302238ce83b4591b3d062ef87517f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:52.197584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:52.209978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1015/2031] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:18:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:52.223454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ebe73-2c12-4dd7-824d-cd0ef5863233 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:52.231278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:52.245670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ebe73-2c12-4dd7-824d-cd0ef5863233 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:52.252439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:52.253291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:52.253679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:52.254275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:52.264081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:52.266738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:52.296211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1017/2032] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:52.298278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:52.299960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:52.407782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8848cedc-43f0-49f3-a4b5-a6a670920da4 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:52.413197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766436-d737-4a51-a855-579677de9229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:52.413862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766436-d737-4a51-a855-579677de9229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:52.414141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766436-d737-4a51-a855-579677de9229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:52.414414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73766436-d737-4a51-a855-579677de9229 admin admin] PATH_INFO: `/v3/users/c32624d142314679bdfe3027832d1b11` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:52.415569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73766436-d737-4a51-a855-579677de9229 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c32624d142314679bdfe3027832d1b11)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:52.417486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73766436-d737-4a51-a855-579677de9229 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:52.469982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c32624d142314679bdfe3027832d1b11'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.470676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c32624d142314679bdfe3027832d1b11'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.478097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c32624d142314679bdfe3027832d1b11'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.478790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c32624d142314679bdfe3027832d1b11'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.486454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c32624d142314679bdfe3027832d1b11'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.497565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:52.506870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1016/2033] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:52 2022] DELETE /identity/v3/users/c32624d142314679bdfe3027832d1b11 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:52.557430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:52.570661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b6656-eb84-4f02-88b3-d9578070b752 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:52.640164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:52.640799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:52.641130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:52.641637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:52.643718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:52.732002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:52.735677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:52.777400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:52.778856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:52.788977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1018/2034] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:52 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:52.822010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73766436-d737-4a51-a855-579677de9229 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:52.827027 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:52.827668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:52.827957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:52.828572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] PATH_INFO: `/v3/projects/aa03cfd05b6a4245ab91f2173cebb615` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:52.829802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa03cfd05b6a4245ab91f2173cebb615)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:52.831735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:52.880152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa03cfd05b6a4245ab91f2173cebb615', 'request_id': 'req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.890866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa03cfd05b6a4245ab91f2173cebb615', 'request_id': 'req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.900860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa03cfd05b6a4245ab91f2173cebb615', 'request_id': 'req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.917185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] The token cache is being invalidate because project aa03cfd05b6a4245ab91f2173cebb615 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:52.917475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.918716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:52.925275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1017/2035] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:52 2022] DELETE /identity/v3/projects/aa03cfd05b6a4245ab91f2173cebb615 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:52.925808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:52.938642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d657a6-f311-4aff-95aa-c7d97db39c25 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.027203 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.027848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.028158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.028434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] PATH_INFO: `/v3/users/6930512abf8d4fdcadbf059ec4c4149f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.029601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6930512abf8d4fdcadbf059ec4c4149f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.031987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.122850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:53.126074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6930512abf8d4fdcadbf059ec4c4149f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.136334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6930512abf8d4fdcadbf059ec4c4149f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.137145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6930512abf8d4fdcadbf059ec4c4149f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.137777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6930512abf8d4fdcadbf059ec4c4149f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.150251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9c9c9baf-0e42-4c0a-a2f0-89069e1ce32a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6930512abf8d4fdcadbf059ec4c4149f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.188300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1019/2036] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:18:52 2022] DELETE /identity/v3/users/6930512abf8d4fdcadbf059ec4c4149f => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:18:53.370290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:53.371778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:53.385082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d4ddc7-0b3a-4e10-be89-799bf5f7c803 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.446602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.447156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.447430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.447699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] PATH_INFO: `/v3/projects/a8dc6a43c1f64c3c9d64b669e1ea93ea` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.448842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8dc6a43c1f64c3c9d64b669e1ea93ea)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.450839 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.483096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:53.505179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8dc6a43c1f64c3c9d64b669e1ea93ea', 'request_id': 'req-dce628bf-037b-4752-8339-e05145c55aeb'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.511602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8dc6a43c1f64c3c9d64b669e1ea93ea', 'request_id': 'req-dce628bf-037b-4752-8339-e05145c55aeb'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.524687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8dc6a43c1f64c3c9d64b669e1ea93ea', 'request_id': 'req-dce628bf-037b-4752-8339-e05145c55aeb'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.537404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] The token cache is being invalidate because project a8dc6a43c1f64c3c9d64b669e1ea93ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:18:53.537694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.538329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-dce628bf-037b-4752-8339-e05145c55aeb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:18:53.539807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1018/2037] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:18:53 2022] DELETE /identity/v3/projects/a8dc6a43c1f64c3c9d64b669e1ea93ea => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:53.544865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f2a2b-8916-430f-b7fe-d8bf88197499 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.545851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f2a2b-8916-430f-b7fe-d8bf88197499 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.546135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f2a2b-8916-430f-b7fe-d8bf88197499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.567804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1a7f2a2b-8916-430f-b7fe-d8bf88197499 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:53.569084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1a7f2a2b-8916-430f-b7fe-d8bf88197499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:53.627841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1020/2038] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:18:53.635746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cdbe8-7abb-41f2-b13e-496760982288 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.636028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cdbe8-7abb-41f2-b13e-496760982288 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.636297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68cdbe8-7abb-41f2-b13e-496760982288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.656431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b68cdbe8-7abb-41f2-b13e-496760982288 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:53.658016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b68cdbe8-7abb-41f2-b13e-496760982288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:53.665957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1019/2039] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:53.670765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f2a2b-8916-430f-b7fe-d8bf88197499 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:53.680355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f2a2b-8916-430f-b7fe-d8bf88197499 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.684891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.685547 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.686401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.686667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.687315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:18:53.687620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.689268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.700562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1021/2040] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:18:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:18:53.704654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68cdbe8-7abb-41f2-b13e-496760982288 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:53.713901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68cdbe8-7abb-41f2-b13e-496760982288 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.718809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.719544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.719828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.720107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.720750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.722671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.755021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1020/2041] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:18:53 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:18:53.759112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:53.769887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ff660d-eb7e-425c-86dc-23d4412fb642 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.833823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.834567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.834715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.834973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.835611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.837597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.861357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1022/2042] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:18:53 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:18:53.865476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:53.876015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aace1a1d-e5c4-445e-8851-3279eeff4ba3 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.880800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.881509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.881803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.882083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.882762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.884353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.895038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1021/2043] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:18:53.899086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:53.909810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f9b8ff-1ec7-41df-9d8c-b0304c9c494f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.914525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6f1eba-d810-4820-a446-a5f1dcaee0c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.915133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6f1eba-d810-4820-a446-a5f1dcaee0c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.915463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6f1eba-d810-4820-a446-a5f1dcaee0c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.915693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6f1eba-d810-4820-a446-a5f1dcaee0c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.916574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6f1eba-d810-4820-a446-a5f1dcaee0c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.918212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6f1eba-d810-4820-a446-a5f1dcaee0c6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.932233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1023/2044] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:18:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:18:53.939044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:53.949227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff026d17-b234-4026-9b3a-f8b4c3a253ce admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:53.954870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:53.955464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:53.955744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:53.956019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] PATH_INFO: `/v3/projects/bb754458f9dd4e2494de1f45973ab341/users/7010544b41374ddd8c8c322fc2a80ec8/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:53.972873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb754458f9dd4e2494de1f45973ab341, user_id=7010544b41374ddd8c8c322fc2a80ec8, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:53.975636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:53.989389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1022/2045] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:18:53 2022] PUT /identity/v3/projects/bb754458f9dd4e2494de1f45973ab341/users/7010544b41374ddd8c8c322fc2a80ec8/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:18:54.001847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb566c-56a0-4e20-948e-e812a591dd63 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:54.002136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb566c-56a0-4e20-948e-e812a591dd63 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:54.002435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb566c-56a0-4e20-948e-e812a591dd63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:54.031357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7edb566c-56a0-4e20-948e-e812a591dd63 None None] MFA Rules not processed for user `7010544b41374ddd8c8c322fc2a80ec8`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:18:54.032560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7edb566c-56a0-4e20-948e-e812a591dd63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:54.048959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:18:54.098612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1024/2046] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:18:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:18:54.110991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:54.112673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:54.174000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf59255-5f00-4fb1-8cdf-1352783287f5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:54.178136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:54.178888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:54.179303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:54.179560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:54.189019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:54.190441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:54.199336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1023/2047] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:18:57.333847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edb566c-56a0-4e20-948e-e812a591dd63 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:18:57.344820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edb566c-56a0-4e20-948e-e812a591dd63 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:18:57.348610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924305c7-7aac-4223-8e28-b132c6ac1abe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:18:57.349362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924305c7-7aac-4223-8e28-b132c6ac1abe service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:18:57.349702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924305c7-7aac-4223-8e28-b132c6ac1abe service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:18:57.350036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924305c7-7aac-4223-8e28-b132c6ac1abe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:18:57.351816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-924305c7-7aac-4223-8e28-b132c6ac1abe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:18:57.415595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924305c7-7aac-4223-8e28-b132c6ac1abe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:18:57.417340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924305c7-7aac-4223-8e28-b132c6ac1abe service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:18:57.427395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1025/2048] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:18:57 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:00.897873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:00.899378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:00.950859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:00.964917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097fa405-527a-41bf-8e9f-771de8e9f515 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:00.970347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:00.971015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:00.971303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:00.971629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] PATH_INFO: `/v3/users/7b0b575aca894f6ba3f7dc9b3ae5b87d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:00.972795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b0b575aca894f6ba3f7dc9b3ae5b87d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:00.974807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:01.022977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b0b575aca894f6ba3f7dc9b3ae5b87d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.023655 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b0b575aca894f6ba3f7dc9b3ae5b87d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.031605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b0b575aca894f6ba3f7dc9b3ae5b87d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.032197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b0b575aca894f6ba3f7dc9b3ae5b87d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.039033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b0b575aca894f6ba3f7dc9b3ae5b87d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.056854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1024/2049] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:00 2022] DELETE /identity/v3/users/7b0b575aca894f6ba3f7dc9b3ae5b87d => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:01.061911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:01.062432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:01.062533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:01.081107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:01.082309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:01.155569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1026/2050] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:01.165470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:01.166920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:01.226340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a531140b-6f7e-46ec-9745-e6ac1a7c6ec9 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:01.229938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:01.230672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:01.230801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:01.231070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:01.241602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:01.243317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:01.256263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1025/2051] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:01.593929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:01.595403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:01.611645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6b0016-8b98-4e17-9ff5-991a39424e5e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:01.617496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:01.618138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:01.618520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:01.618866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] PATH_INFO: `/v3/projects/736dec385e1b40a5a780fed18a76bae2` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:01.620022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=736dec385e1b40a5a780fed18a76bae2)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:01.622124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:01.675085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '736dec385e1b40a5a780fed18a76bae2', 'request_id': 'req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.677720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:01.683518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '736dec385e1b40a5a780fed18a76bae2', 'request_id': 'req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.691804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '736dec385e1b40a5a780fed18a76bae2', 'request_id': 'req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.703055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] The token cache is being invalidate because project 736dec385e1b40a5a780fed18a76bae2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:01.703354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.704025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.705544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1027/2052] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:01 2022] DELETE /identity/v3/projects/736dec385e1b40a5a780fed18a76bae2 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:01.709524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:01.719743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b92292-877b-4416-90a7-0ed5d77bf4d7 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:01.778457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:01.779130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:01.779410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:01.779685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] PATH_INFO: `/v3/users/b6c0713f0417492ca2ab6fcc1312fa64` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:01.780841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6c0713f0417492ca2ab6fcc1312fa64)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:01.782837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:01.799131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:01.832883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6c0713f0417492ca2ab6fcc1312fa64'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.833633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6c0713f0417492ca2ab6fcc1312fa64'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.842298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6c0713f0417492ca2ab6fcc1312fa64'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.843653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6c0713f0417492ca2ab6fcc1312fa64'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.855895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6c0713f0417492ca2ab6fcc1312fa64'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:01.880787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1026/2053] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:01 2022] DELETE /identity/v3/users/b6c0713f0417492ca2ab6fcc1312fa64 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:02.264686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:02.266132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:02.283960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ff2205-defa-4dd9-b8af-29c94a6ebca4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:02.348368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:02.349030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:02.349308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:02.349577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] PATH_INFO: `/v3/projects/061e5ad9426c4a9cb1eeef269dfc6f42` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:02.350668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] RBAC: Authorizing `identity:delete_project(project_id=061e5ad9426c4a9cb1eeef269dfc6f42)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:02.352559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:02.396600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '061e5ad9426c4a9cb1eeef269dfc6f42', 'request_id': 'req-fa53edf2-a032-4466-b59a-bf20debf7795'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.403681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '061e5ad9426c4a9cb1eeef269dfc6f42', 'request_id': 'req-fa53edf2-a032-4466-b59a-bf20debf7795'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.410959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '061e5ad9426c4a9cb1eeef269dfc6f42', 'request_id': 'req-fa53edf2-a032-4466-b59a-bf20debf7795'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.421970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] The token cache is being invalidate because project 061e5ad9426c4a9cb1eeef269dfc6f42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:02.422275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.422915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.424394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1028/2054] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:02 2022] DELETE /identity/v3/projects/061e5ad9426c4a9cb1eeef269dfc6f42 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:02.428655 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:02.430305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:02.444834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c14b7e-5384-4e18-99d6-065a49ea492a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:02.514729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:02.515366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:02.515886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:02.516167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] PATH_INFO: `/v3/users/01fdb5f4ff854e13bf802348b54b73f4` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:02.517423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] RBAC: Authorizing `identity:delete_user(user_id=01fdb5f4ff854e13bf802348b54b73f4)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:02.519344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:02.574600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01fdb5f4ff854e13bf802348b54b73f4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.575257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01fdb5f4ff854e13bf802348b54b73f4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.583604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01fdb5f4ff854e13bf802348b54b73f4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.584731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01fdb5f4ff854e13bf802348b54b73f4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.593771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ec26c39-b8c6-4e6c-97ea-e756126d579f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01fdb5f4ff854e13bf802348b54b73f4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:02.613873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1027/2055] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:02 2022] DELETE /identity/v3/users/01fdb5f4ff854e13bf802348b54b73f4 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:02.925036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:02.926448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:02.950687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa53edf2-a032-4466-b59a-bf20debf7795 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.013801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.014425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.014705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.014982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] PATH_INFO: `/v3/projects/e331719e89fc43838931b41bbf397864` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.016095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e331719e89fc43838931b41bbf397864)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.018039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.029873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:03.064373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e331719e89fc43838931b41bbf397864', 'request_id': 'req-4a76c204-7f42-4814-9f73-8eb121d1287c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:03.070864 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e331719e89fc43838931b41bbf397864', 'request_id': 'req-4a76c204-7f42-4814-9f73-8eb121d1287c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:03.077358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e331719e89fc43838931b41bbf397864', 'request_id': 'req-4a76c204-7f42-4814-9f73-8eb121d1287c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:03.088031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] The token cache is being invalidate because project e331719e89fc43838931b41bbf397864 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:03.088259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:03.088896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a76c204-7f42-4814-9f73-8eb121d1287c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:03.090422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1029/2056] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:02 2022] DELETE /identity/v3/projects/e331719e89fc43838931b41bbf397864 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:03.096960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3979b5e-e046-4cdc-87bb-28224eb2f77c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.097256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3979b5e-e046-4cdc-87bb-28224eb2f77c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.097530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3979b5e-e046-4cdc-87bb-28224eb2f77c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.117674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e3979b5e-e046-4cdc-87bb-28224eb2f77c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:03.118867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e3979b5e-e046-4cdc-87bb-28224eb2f77c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:03.127300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:03.182245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1028/2057] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:03.191630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2bc0ff-43fd-4f6f-8684-6d50fe858c1e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.191913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2bc0ff-43fd-4f6f-8684-6d50fe858c1e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.192189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2bc0ff-43fd-4f6f-8684-6d50fe858c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.218252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5b2bc0ff-43fd-4f6f-8684-6d50fe858c1e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:03.219488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5b2bc0ff-43fd-4f6f-8684-6d50fe858c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:03.228610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1030/2058] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:03.232549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3979b5e-e046-4cdc-87bb-28224eb2f77c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.242588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3979b5e-e046-4cdc-87bb-28224eb2f77c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.247409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.248551 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.248828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.249149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.249808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:03.250111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.251692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.265704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1029/2059] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:03.270831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2bc0ff-43fd-4f6f-8684-6d50fe858c1e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.282421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2bc0ff-43fd-4f6f-8684-6d50fe858c1e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.292742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.293642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.293990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.294329 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.295792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.297972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.328659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1031/2060] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:03 2022] POST /identity/v3/projects => generated 391 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:03.333202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.344019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4969698-e1f3-4db0-aaba-d2dd76dd61f1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.405080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.405757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.406039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.406310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.406946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.408861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.430327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1030/2061] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:03 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:03.434832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.445322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49ce13e-b5aa-4e3a-95a8-273f48d1a781 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.449639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.450231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.451241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.451520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.452257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.453929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.463946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1032/2062] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:03.468714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.479992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b89c8f5-dc0d-43f9-9988-507b6f139dfc admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.486735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c00656f-f501-40d6-a98e-bfb9cbf00bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.487412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c00656f-f501-40d6-a98e-bfb9cbf00bd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.487760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c00656f-f501-40d6-a98e-bfb9cbf00bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.488096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c00656f-f501-40d6-a98e-bfb9cbf00bd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.488814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c00656f-f501-40d6-a98e-bfb9cbf00bd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.490536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c00656f-f501-40d6-a98e-bfb9cbf00bd6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.518988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1031/2063] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:03.527754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.540459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d44c509-ff64-45c5-a153-66a864a2cab9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.545562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.546380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.546660 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.546961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] PATH_INFO: `/v3/projects/95d7b263936e4e5d965ab90fb4363913/users/a2f3d6ea1fb842a0878f089229eb066e/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.563646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] RBAC: Authorizing `identity:create_grant(project_id=95d7b263936e4e5d965ab90fb4363913, user_id=a2f3d6ea1fb842a0878f089229eb066e, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.566546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.580908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1033/2064] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:03 2022] PUT /identity/v3/projects/95d7b263936e4e5d965ab90fb4363913/users/a2f3d6ea1fb842a0878f089229eb066e/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:03.593096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a8fd2-0801-4257-bd8b-6e58d2a54b11 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.593390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a8fd2-0801-4257-bd8b-6e58d2a54b11 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.593801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a8fd2-0801-4257-bd8b-6e58d2a54b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.621184 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ad6a8fd2-0801-4257-bd8b-6e58d2a54b11 None None] MFA Rules not processed for user `a2f3d6ea1fb842a0878f089229eb066e`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:03.622321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ad6a8fd2-0801-4257-bd8b-6e58d2a54b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:03.686366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1032/2065] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:03 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:03.690827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.752191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dff0423-eed3-456e-af1b-98f301ec874d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.757024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.757627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.757911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.758193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.758861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.760827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.788739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1034/2066] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:03 2022] POST /identity/v3/projects => generated 391 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:03.792827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a8fd2-0801-4257-bd8b-6e58d2a54b11 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.804375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a8fd2-0801-4257-bd8b-6e58d2a54b11 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.809290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:03.865072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.865687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.865976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.866247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.866875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.868837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.892673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1033/2067] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:03 2022] POST /identity/v3/users => generated 302 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:03.896739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.907640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1b1436-c969-444d-9cc2-d663764b9205 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.912259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.912949 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.913317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.913587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.914352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.915934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.930407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1035/2068] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:03.932443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.943460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0f77fc-16f2-4b86-b9f4-9b67641e6ae5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:03.948357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:03.949178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:03.949482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:03.949744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] PATH_INFO: `/v3/projects/b9a12e1ff3af45e58b898e49126b9854/users/83f83c3e5898401bb96dedbe287d7a49/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:03.965207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9a12e1ff3af45e58b898e49126b9854, user_id=83f83c3e5898401bb96dedbe287d7a49, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:03.968443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:03.982991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1034/2069] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:03 2022] PUT /identity/v3/projects/b9a12e1ff3af45e58b898e49126b9854/users/83f83c3e5898401bb96dedbe287d7a49/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:03.987703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:03.999786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d0b3bb-9d6e-49cf-9fdd-ef9ce4984f1a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:04.008041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:04.060302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f847370-35ed-4d2a-93ff-cf899db01d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:04.061121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f847370-35ed-4d2a-93ff-cf899db01d4e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:04.061406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f847370-35ed-4d2a-93ff-cf899db01d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:04.061685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f847370-35ed-4d2a-93ff-cf899db01d4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:04.062333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f847370-35ed-4d2a-93ff-cf899db01d4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:04.063938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f847370-35ed-4d2a-93ff-cf899db01d4e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:04.073495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1036/2070] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:04.078512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:04.088628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33d7487-c26d-4a6a-969b-17f3606e1ff9 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:04.094018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:04.094661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:04.094941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:04.095224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] PATH_INFO: `/v3/projects/b9a12e1ff3af45e58b898e49126b9854/users/83f83c3e5898401bb96dedbe287d7a49/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:04.097122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9a12e1ff3af45e58b898e49126b9854, user_id=83f83c3e5898401bb96dedbe287d7a49, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:04.099761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:04.109900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1035/2071] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:04 2022] PUT /identity/v3/projects/b9a12e1ff3af45e58b898e49126b9854/users/83f83c3e5898401bb96dedbe287d7a49/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:04.119922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:04.120214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:04.120494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:04.149320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] MFA Rules not processed for user `83f83c3e5898401bb96dedbe287d7a49`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:04.150469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:04.222225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1037/2072] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:04 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:04.234113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:04.235885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:04.304195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f9b1fc-a66d-404b-86e9-77dfabbe275f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:04.307907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:04.308530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:04.308807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:04.309113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:04.319327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:04.320890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:04.331676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1036/2073] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:04 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:04.379587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:04.381062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:04.393591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5bc8cd-842b-42ba-a0ca-3746fe6577a6 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:04.397370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19b2807-511a-402c-ae4c-0263b8df26ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:04.397980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b2807-511a-402c-ae4c-0263b8df26ac service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:04.398263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b2807-511a-402c-ae4c-0263b8df26ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:04.398534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b2807-511a-402c-ae4c-0263b8df26ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:04.457605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19b2807-511a-402c-ae4c-0263b8df26ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:04.459044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19b2807-511a-402c-ae4c-0263b8df26ac service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:04.468796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1038/2074] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:04 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:06.093656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:06.102679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:06.250131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d02211-cfc2-48b8-9a4c-7a0cb2faaba2 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:06.260624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:06.261493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:06.261841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:06.262174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] PATH_INFO: `/v3/users/399c3f71fb5d495a9c3a7148f8357bde` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:06.264216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] RBAC: Authorizing `identity:delete_user(user_id=399c3f71fb5d495a9c3a7148f8357bde)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:06.268973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:06.274457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:06.324085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '399c3f71fb5d495a9c3a7148f8357bde'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:06.324803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '399c3f71fb5d495a9c3a7148f8357bde'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:06.336310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '399c3f71fb5d495a9c3a7148f8357bde'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:06.342485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '399c3f71fb5d495a9c3a7148f8357bde'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:06.352087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '399c3f71fb5d495a9c3a7148f8357bde'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:06.374377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1037/2075] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:06 2022] DELETE /identity/v3/users/399c3f71fb5d495a9c3a7148f8357bde => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:06.379716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:06.380184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:06.380326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:06.402169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:06.403383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:06.430914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:06.473552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1039/2076] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:06.482125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:06.483650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:06.543545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b95fd-2d28-404d-bb5d-b5e2dc474ded admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:06.547935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95de14ac-0038-4644-9a44-490d0b2d6ec1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:06.548540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95de14ac-0038-4644-9a44-490d0b2d6ec1 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:06.548820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95de14ac-0038-4644-9a44-490d0b2d6ec1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:06.549128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95de14ac-0038-4644-9a44-490d0b2d6ec1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:06.558542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95de14ac-0038-4644-9a44-490d0b2d6ec1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:06.560603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95de14ac-0038-4644-9a44-490d0b2d6ec1 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:06.570640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1038/2077] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:07.075263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.077213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:07.102213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d044ae91-ff87-4f34-b488-d5082d11efeb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:07.107723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:07.108393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.108671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.109034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] PATH_INFO: `/v3/projects/c4f435e89b704bb0a5fd4b6373f5e1a3` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.110148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4f435e89b704bb0a5fd4b6373f5e1a3)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:07.112838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:07.166659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4f435e89b704bb0a5fd4b6373f5e1a3', 'request_id': 'req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:07.175387 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4f435e89b704bb0a5fd4b6373f5e1a3', 'request_id': 'req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:07.182620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4f435e89b704bb0a5fd4b6373f5e1a3', 'request_id': 'req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:07.194536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] The token cache is being invalidate because project c4f435e89b704bb0a5fd4b6373f5e1a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:07.194815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:07.195463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a2ea2ce-aab2-4edd-a5cb-25b7b3229819 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:07.197020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1040/2078] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:07 2022] DELETE /identity/v3/projects/c4f435e89b704bb0a5fd4b6373f5e1a3 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:07.202255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b5bb77-f082-441d-9637-8af87d937f0b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.202735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b5bb77-f082-441d-9637-8af87d937f0b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.202838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b5bb77-f082-441d-9637-8af87d937f0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.222610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-46b5bb77-f082-441d-9637-8af87d937f0b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:07.223811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-46b5bb77-f082-441d-9637-8af87d937f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:07.291426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1039/2079] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:07.301237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f950859e-576a-498f-9bbf-147d86fa2274 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.301525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f950859e-576a-498f-9bbf-147d86fa2274 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.301807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f950859e-576a-498f-9bbf-147d86fa2274 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.331186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f950859e-576a-498f-9bbf-147d86fa2274 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:07.332722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f950859e-576a-498f-9bbf-147d86fa2274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:07.344605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1041/2080] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:07.348271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b5bb77-f082-441d-9637-8af87d937f0b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.367211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b5bb77-f082-441d-9637-8af87d937f0b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:07.376051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:07.377017 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.377328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.377627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.378437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:07.378755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:07.380702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:07.401073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1040/2081] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:07.403334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f950859e-576a-498f-9bbf-147d86fa2274 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.414601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f950859e-576a-498f-9bbf-147d86fa2274 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:07.420135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:07.420859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.421298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.421572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.422384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:07.424464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:07.473780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1042/2082] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:07 2022] POST /identity/v3/projects => generated 361 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:07.478124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.490108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071e0086-8474-40c1-8662-7b11fea20ec7 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:07.500686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:07.566813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:07.567212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.567521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.567781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.568713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:07.571046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:07.596944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1041/2083] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:07 2022] POST /identity/v3/users => generated 281 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:07.598050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.615579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8600-e3e8-4001-a9f3-505a52a1db4b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:07.620921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:07.621621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.621967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.622302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.625330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:07.626987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:07.637985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1043/2084] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:07 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:07.642670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.655093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc5defd-554e-414a-9ccc-c959c1ddce23 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:07.660491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c70f92f-3615-4406-9e2b-6f0827f04968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:07.661269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c70f92f-3615-4406-9e2b-6f0827f04968 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.661638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c70f92f-3615-4406-9e2b-6f0827f04968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.662037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c70f92f-3615-4406-9e2b-6f0827f04968 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.662801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c70f92f-3615-4406-9e2b-6f0827f04968 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:07.665133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c70f92f-3615-4406-9e2b-6f0827f04968 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:07.680716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1042/2085] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:07 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:07.684807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.696435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9795ce-bb36-4fbd-bff5-a34260ba888b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:07.702150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:07.702694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.703097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.703369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] PATH_INFO: `/v3/projects/aaae9e44e26f412996da9f9a5df015a4/users/46abdd0c7e1548a28a0f1f924942d42e/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.720952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaae9e44e26f412996da9f9a5df015a4, user_id=46abdd0c7e1548a28a0f1f924942d42e, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:07.723735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:07.738768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1044/2086] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:07 2022] PUT /identity/v3/projects/aaae9e44e26f412996da9f9a5df015a4/users/46abdd0c7e1548a28a0f1f924942d42e/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:07.749965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57b5935-c1f3-4deb-9bae-f02e519800b6 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:07.750353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57b5935-c1f3-4deb-9bae-f02e519800b6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:07.750720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57b5935-c1f3-4deb-9bae-f02e519800b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:07.783108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d57b5935-c1f3-4deb-9bae-f02e519800b6 None None] MFA Rules not processed for user `46abdd0c7e1548a28a0f1f924942d42e`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:07.784697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d57b5935-c1f3-4deb-9bae-f02e519800b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:07.883285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1043/2087] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:07 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:07.946443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:07.948466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:08.031728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1a8b04-3263-4d11-923a-8b240fcd8007 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:08.035391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f390458b-7941-4289-8a64-c2974968e542 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:08.036062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f390458b-7941-4289-8a64-c2974968e542 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:08.036353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f390458b-7941-4289-8a64-c2974968e542 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:08.036631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f390458b-7941-4289-8a64-c2974968e542 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:08.042186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:08.048590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f390458b-7941-4289-8a64-c2974968e542 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:08.050884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f390458b-7941-4289-8a64-c2974968e542 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:08.072704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1045/2088] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:07 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:08.090954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57b5935-c1f3-4deb-9bae-f02e519800b6 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:08.181700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57b5935-c1f3-4deb-9bae-f02e519800b6 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:08.190003 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:08.191332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:08.192045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:08.192656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] PATH_INFO: `/v3/users/46abdd0c7e1548a28a0f1f924942d42e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:08.194595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46abdd0c7e1548a28a0f1f924942d42e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:08.198126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:08.236854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:08.296343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46abdd0c7e1548a28a0f1f924942d42e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:08.300055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46abdd0c7e1548a28a0f1f924942d42e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:08.323204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46abdd0c7e1548a28a0f1f924942d42e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:08.324195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46abdd0c7e1548a28a0f1f924942d42e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:08.342261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46abdd0c7e1548a28a0f1f924942d42e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:08.384044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1044/2089] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:08 2022] DELETE /identity/v3/users/46abdd0c7e1548a28a0f1f924942d42e => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:08.394888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:08.395181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:08.395479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:08.420622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:08.421896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:08.488385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1046/2090] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:08.498463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:08.499842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:08.563378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e565cfd-1068-4358-808c-e1b5c9df4f38 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:08.567021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044d35aa-91de-4bd7-9206-5c775a5b1346 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:08.567623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d35aa-91de-4bd7-9206-5c775a5b1346 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:08.568016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d35aa-91de-4bd7-9206-5c775a5b1346 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:08.568156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044d35aa-91de-4bd7-9206-5c775a5b1346 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:08.579948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044d35aa-91de-4bd7-9206-5c775a5b1346 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:08.581846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044d35aa-91de-4bd7-9206-5c775a5b1346 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:08.596399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1045/2091] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:19:08.923236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:08.925895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:08.940916 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c602044c-1397-477a-989d-af3c537e9e65 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:08.945769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:08.946481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:08.946860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:08.947463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] PATH_INFO: `/v3/projects/aaae9e44e26f412996da9f9a5df015a4` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:08.949490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aaae9e44e26f412996da9f9a5df015a4)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:08.952177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.002282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aaae9e44e26f412996da9f9a5df015a4', 'request_id': 'req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.011880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aaae9e44e26f412996da9f9a5df015a4', 'request_id': 'req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.017636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aaae9e44e26f412996da9f9a5df015a4', 'request_id': 'req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.029712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] The token cache is being invalidate because project aaae9e44e26f412996da9f9a5df015a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:09.030013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.030690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-21e8cc56-ebb0-4181-805e-7c3d0b8e04d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.032201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1047/2092] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:08 2022] DELETE /identity/v3/projects/aaae9e44e26f412996da9f9a5df015a4 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:09.037488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.037782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.038088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.058527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:09.059737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.128994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1046/2093] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:09.134516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef067cc6-3e27-4d9b-83b4-9c7950bf9c57 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.134815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef067cc6-3e27-4d9b-83b4-9c7950bf9c57 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.135138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef067cc6-3e27-4d9b-83b4-9c7950bf9c57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.156292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ef067cc6-3e27-4d9b-83b4-9c7950bf9c57 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:09.157558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ef067cc6-3e27-4d9b-83b4-9c7950bf9c57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.166309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1048/2094] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:09.170030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.170835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.171098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef067cc6-3e27-4d9b-83b4-9c7950bf9c57 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.181215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef067cc6-3e27-4d9b-83b4-9c7950bf9c57 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.186420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1fb3d8-40d8-4700-a02d-326fa5ed78c0 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.187844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.188693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.189068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.189355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.190069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:09.190383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.192626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.193894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.194604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.194902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.195179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] PATH_INFO: `/v3/users/a2f3d6ea1fb842a0878f089229eb066e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.196541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2f3d6ea1fb842a0878f089229eb066e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.198612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.206969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1049/2095] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:19:09.211315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.223757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30597be7-84e4-4275-875f-6ad078a73100 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.228675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.229509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.229797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.230076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.231184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.233488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.243528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:09.258333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2f3d6ea1fb842a0878f089229eb066e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.259577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2f3d6ea1fb842a0878f089229eb066e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.266291 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1050/2096] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:09 2022] POST /identity/v3/projects => generated 365 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:09.268783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2f3d6ea1fb842a0878f089229eb066e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.270142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2f3d6ea1fb842a0878f089229eb066e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.271583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.273315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.278669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-57a6104f-f456-46cc-9567-e209cc40cbd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2f3d6ea1fb842a0878f089229eb066e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:09.297708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1047/2097] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:09 2022] DELETE /identity/v3/users/a2f3d6ea1fb842a0878f089229eb066e => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:09.302180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.302459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.302735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.315019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:09.322628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:09.323838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.343141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf0a894-329a-4d18-9dee-9c923e661a53 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.382663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1048/2098] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:09.397621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.399163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.407156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.407792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.408073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.408346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.409016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.411360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.438813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1051/2099] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:09 2022] POST /identity/v3/users => generated 283 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:09.439926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.462789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b6e78f-bbad-4d1c-a8d9-8ae44107b1a0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.462789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.462789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.462789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.463120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.463120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.463120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.473150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1052/2100] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:09.474366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.486125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2fe236-146d-49ec-b85e-30e70c1c26a3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.490517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6b9af5-a00f-4a27-ba22-0cb23cd31078 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.491240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.492052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.492440 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.492754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.493404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.495135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.496103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb4200-4a5e-4feb-bb51-b8346588d249 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.496829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb4200-4a5e-4feb-bb51-b8346588d249 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.497240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb4200-4a5e-4feb-bb51-b8346588d249 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.498229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb4200-4a5e-4feb-bb51-b8346588d249 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.507783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1053/2101] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:09.511977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.519962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fb4200-4a5e-4feb-bb51-b8346588d249 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.523851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fb4200-4a5e-4feb-bb51-b8346588d249 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.524211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e48a4b-9e33-4e2c-ace3-70139d465a63 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.529525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.530327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.530981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.531274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] PATH_INFO: `/v3/projects/f8b249a356c44057aa75318ff1ec4b1d/users/de6cb1890f484838bab3f3b662d82606/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.548851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1049/2102] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:09.555579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8b249a356c44057aa75318ff1ec4b1d, user_id=de6cb1890f484838bab3f3b662d82606, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.558456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.576368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1054/2103] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:09 2022] PUT /identity/v3/projects/f8b249a356c44057aa75318ff1ec4b1d/users/de6cb1890f484838bab3f3b662d82606/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:09.586748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.587047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.587332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.618142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] MFA Rules not processed for user `de6cb1890f484838bab3f3b662d82606`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:09.619369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.695444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:09.739631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1050/2104] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:09 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:09.748211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.750232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:09.809427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71098c75-6007-4463-8557-1ea6ba33e847 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:09.814685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:09.815394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:09.815750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:09.816087 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:09.826163 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:09.827643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:09.836869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1055/2105] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:09 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:09.934550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:09.937077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:10.030530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8052a4-5761-4927-aecf-9d7061860609 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:10.041818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:10.042795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:10.043144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:10.043498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] PATH_INFO: `/v3/projects/95d7b263936e4e5d965ab90fb4363913` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:10.045366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95d7b263936e4e5d965ab90fb4363913)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:10.047873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:10.104678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95d7b263936e4e5d965ab90fb4363913', 'request_id': 'req-40f3f0be-441a-4a11-ab06-00e6014303a2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.112285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95d7b263936e4e5d965ab90fb4363913', 'request_id': 'req-40f3f0be-441a-4a11-ab06-00e6014303a2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.122024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95d7b263936e4e5d965ab90fb4363913', 'request_id': 'req-40f3f0be-441a-4a11-ab06-00e6014303a2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.135306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] The token cache is being invalidate because project 95d7b263936e4e5d965ab90fb4363913 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:10.135601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.136276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.141422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1051/2106] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:09 2022] DELETE /identity/v3/projects/95d7b263936e4e5d965ab90fb4363913 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:10.142418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:10.164141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a5295b-a64e-4d0e-ba86-d96bda571270 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:10.197991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:10.246604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:10.247669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:10.247964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:10.248238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] PATH_INFO: `/v3/users/83f83c3e5898401bb96dedbe287d7a49` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:10.249832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] RBAC: Authorizing `identity:delete_user(user_id=83f83c3e5898401bb96dedbe287d7a49)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:10.251757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:10.300782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83f83c3e5898401bb96dedbe287d7a49'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.309409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83f83c3e5898401bb96dedbe287d7a49'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.310699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83f83c3e5898401bb96dedbe287d7a49'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.311331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83f83c3e5898401bb96dedbe287d7a49'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.318887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b9be90c1-f351-42e2-976b-9cb69a38a319 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83f83c3e5898401bb96dedbe287d7a49'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.338009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1056/2107] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:10 2022] DELETE /identity/v3/users/83f83c3e5898401bb96dedbe287d7a49 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:10.618863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:10.620464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:10.634572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f3f0be-441a-4a11-ab06-00e6014303a2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:10.699877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:10.700527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:10.701488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:10.701765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] PATH_INFO: `/v3/projects/b9a12e1ff3af45e58b898e49126b9854` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:10.702900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9a12e1ff3af45e58b898e49126b9854)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:10.704832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:10.728656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:10.760252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9a12e1ff3af45e58b898e49126b9854', 'request_id': 'req-09bad079-659c-4a14-a782-82ab34be786b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.766905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9a12e1ff3af45e58b898e49126b9854', 'request_id': 'req-09bad079-659c-4a14-a782-82ab34be786b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.775445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9a12e1ff3af45e58b898e49126b9854', 'request_id': 'req-09bad079-659c-4a14-a782-82ab34be786b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.788163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] The token cache is being invalidate because project b9a12e1ff3af45e58b898e49126b9854 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:10.788534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.789271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-09bad079-659c-4a14-a782-82ab34be786b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:10.791101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1052/2108] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:10 2022] DELETE /identity/v3/projects/b9a12e1ff3af45e58b898e49126b9854 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:10.796363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99753e-0c27-4b75-a1ee-2be9a19e0a05 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:10.796896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99753e-0c27-4b75-a1ee-2be9a19e0a05 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:10.797199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99753e-0c27-4b75-a1ee-2be9a19e0a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:10.817891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7a99753e-0c27-4b75-a1ee-2be9a19e0a05 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:10.819120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7a99753e-0c27-4b75-a1ee-2be9a19e0a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:10.886640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1057/2109] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:10.896024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127c3310-9256-4c14-9f90-740ce65db65e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:10.896328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127c3310-9256-4c14-9f90-740ce65db65e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:10.896603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127c3310-9256-4c14-9f90-740ce65db65e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:10.918403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-127c3310-9256-4c14-9f90-740ce65db65e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:10.919811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-127c3310-9256-4c14-9f90-740ce65db65e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:10.928331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1053/2110] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:10.932554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99753e-0c27-4b75-a1ee-2be9a19e0a05 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:10.943194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99753e-0c27-4b75-a1ee-2be9a19e0a05 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:10.947978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:10.948710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:10.949129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:10.949524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:10.950895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:10.951380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:10.953160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:10.965759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1058/2111] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:10.970566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127c3310-9256-4c14-9f90-740ce65db65e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:10.980154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127c3310-9256-4c14-9f90-740ce65db65e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:10.985183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:10.985801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:10.986077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:10.986344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:10.986997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:10.988911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.032063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1054/2112] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:10 2022] POST /identity/v3/projects => generated 399 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:11.036158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.046907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6456c20e-f38b-451e-952e-e17f1c111e33 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.084228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:11.114852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.115883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.116244 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.116981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.117623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.119580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.141862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1059/2113] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:11 2022] POST /identity/v3/users => generated 300 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:11.146248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.157974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf058b5-9406-4b21-81fd-74386525d515 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.163943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.164580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.164858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.165170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.165823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.167724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.178235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1055/2114] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:11 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:11.182971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.193613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4488cc-52c3-4b66-b48a-77e6eccb9a05 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.198305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896e5842-3c3f-494f-8eb2-2370492d3873 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.198899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e5842-3c3f-494f-8eb2-2370492d3873 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.199177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e5842-3c3f-494f-8eb2-2370492d3873 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.199444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e5842-3c3f-494f-8eb2-2370492d3873 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.200081 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896e5842-3c3f-494f-8eb2-2370492d3873 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.202550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896e5842-3c3f-494f-8eb2-2370492d3873 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.212354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1060/2115] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:11.216553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.226203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61007bcc-0021-4e96-86c2-c60101716f63 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.230842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.231477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.231840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.232157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] PATH_INFO: `/v3/projects/50fb7200a62b4c0a9b0b554c1dfbaf31/users/5bba5a1670bb4bedab9345546f000161/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.248232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] RBAC: Authorizing `identity:create_grant(project_id=50fb7200a62b4c0a9b0b554c1dfbaf31, user_id=5bba5a1670bb4bedab9345546f000161, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.250945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.268174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1056/2116] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:11 2022] PUT /identity/v3/projects/50fb7200a62b4c0a9b0b554c1dfbaf31/users/5bba5a1670bb4bedab9345546f000161/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:11.278610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15f4555-999d-40fc-a1f1-9c2886f6a57e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.278899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15f4555-999d-40fc-a1f1-9c2886f6a57e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.279178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15f4555-999d-40fc-a1f1-9c2886f6a57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.318385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d15f4555-999d-40fc-a1f1-9c2886f6a57e None None] MFA Rules not processed for user `5bba5a1670bb4bedab9345546f000161`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:11.319732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d15f4555-999d-40fc-a1f1-9c2886f6a57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:11.395061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1061/2117] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:11 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:11.396480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.469116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6196ff1c-aac6-49c1-84be-b4135ae3090c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.473811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.474433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.474711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.474984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.475624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.477643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.515698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1057/2118] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:11 2022] POST /identity/v3/projects => generated 399 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:11.517306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15f4555-999d-40fc-a1f1-9c2886f6a57e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.529872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15f4555-999d-40fc-a1f1-9c2886f6a57e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.593910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.594526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.594806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.595080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.595722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.597686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.619885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1062/2119] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:11 2022] POST /identity/v3/users => generated 306 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:11.624588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.634616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b2cfc3-4e7e-4e4a-9ebb-cb7462f5af54 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.639181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.639772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.640047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.640317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.640992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.642568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.654223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1058/2120] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:11.658543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.670563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bea5235-f2fc-4dd9-9c6b-53b4ff6a3b17 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.675968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.676695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.677075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.677413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] PATH_INFO: `/v3/projects/455c549470dc4403bd9476a4443d0370/users/af5a4ee5fc224075af4e6d7d1bf98cbe/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.707574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] RBAC: Authorizing `identity:create_grant(project_id=455c549470dc4403bd9476a4443d0370, user_id=af5a4ee5fc224075af4e6d7d1bf98cbe, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.710868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.726373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1063/2121] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:11 2022] PUT /identity/v3/projects/455c549470dc4403bd9476a4443d0370/users/af5a4ee5fc224075af4e6d7d1bf98cbe/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:11.730573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.740522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dee957-ba08-460a-8205-d8245708a5f6 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.760505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:11.807354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f14a01-354f-49ab-b0e1-8718ea562d91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.808045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f14a01-354f-49ab-b0e1-8718ea562d91 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.808389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f14a01-354f-49ab-b0e1-8718ea562d91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.810056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f14a01-354f-49ab-b0e1-8718ea562d91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.810056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f14a01-354f-49ab-b0e1-8718ea562d91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.811045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f14a01-354f-49ab-b0e1-8718ea562d91 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.822674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1059/2122] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:11 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:11.825847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:11.836310 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eefd176-fab9-4e97-a797-d1662b8d0dcd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:11.840848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:11.842359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.842646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.842919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] PATH_INFO: `/v3/projects/455c549470dc4403bd9476a4443d0370/users/af5a4ee5fc224075af4e6d7d1bf98cbe/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.844819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] RBAC: Authorizing `identity:create_grant(project_id=455c549470dc4403bd9476a4443d0370, user_id=af5a4ee5fc224075af4e6d7d1bf98cbe, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:11.847472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:11.850680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:11.860332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1064/2123] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:11 2022] PUT /identity/v3/projects/455c549470dc4403bd9476a4443d0370/users/af5a4ee5fc224075af4e6d7d1bf98cbe/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:11.871627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1f3e9-97fb-468b-be33-c0b30a8fb2cf None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:11.871919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1f3e9-97fb-468b-be33-c0b30a8fb2cf None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:11.872411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1f3e9-97fb-468b-be33-c0b30a8fb2cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:11.901316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-35a1f3e9-97fb-468b-be33-c0b30a8fb2cf None None] MFA Rules not processed for user `af5a4ee5fc224075af4e6d7d1bf98cbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:11.902530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-35a1f3e9-97fb-468b-be33-c0b30a8fb2cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:11.991891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1060/2124] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:11 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:12.005047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:12.008364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:12.099419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400b2ea2-ff7e-4647-8120-f3e65172b302 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:12.108718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:12.109390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:12.109913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:12.109913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:12.121821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:12.124358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:12.136347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1065/2125] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:12 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:12.230315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a1f3e9-97fb-468b-be33-c0b30a8fb2cf None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:12.243225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a1f3e9-97fb-468b-be33-c0b30a8fb2cf None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:12.265888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:12.269585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:12.269937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:12.270288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:12.339345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:12.340807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:12.351288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1061/2126] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:12 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:14.219419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:14.222063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:14.327891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59163825-ddee-416d-8ba0-1f84211113f7 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:14.332985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:14.333618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:14.333897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:14.334182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] PATH_INFO: `/v3/users/bfcc0e7f557e4334a808f5dfaffe93db` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:14.335639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bfcc0e7f557e4334a808f5dfaffe93db)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:14.337557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:14.386537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bfcc0e7f557e4334a808f5dfaffe93db'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.394782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bfcc0e7f557e4334a808f5dfaffe93db'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.395699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bfcc0e7f557e4334a808f5dfaffe93db'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.396283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bfcc0e7f557e4334a808f5dfaffe93db'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.405359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47d436d1-d46c-4770-a63d-93d81cc4c879 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bfcc0e7f557e4334a808f5dfaffe93db'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.444578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1066/2127] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:14 2022] DELETE /identity/v3/users/bfcc0e7f557e4334a808f5dfaffe93db => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:14.626964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:14.643874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0992c50-1717-4c52-9c2f-536126c3f6cc service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:14.686131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:14.706044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:14.706674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:14.706949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:14.707270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] PATH_INFO: `/v3/projects/c1b321c355e94b8496955f09f547f561` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:14.708474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1b321c355e94b8496955f09f547f561)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:14.710614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:14.758349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1b321c355e94b8496955f09f547f561', 'request_id': 'req-d1b1780d-9f0a-4ece-884d-a51a5207ac31'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.765311 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1b321c355e94b8496955f09f547f561', 'request_id': 'req-d1b1780d-9f0a-4ece-884d-a51a5207ac31'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.772588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1b321c355e94b8496955f09f547f561', 'request_id': 'req-d1b1780d-9f0a-4ece-884d-a51a5207ac31'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.785190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] The token cache is being invalidate because project c1b321c355e94b8496955f09f547f561 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:14.785488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.786774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d1b1780d-9f0a-4ece-884d-a51a5207ac31 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:14.788737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1062/2128] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:14 2022] DELETE /identity/v3/projects/c1b321c355e94b8496955f09f547f561 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:14.794152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec4cd67-b7fd-464f-9490-6332f86d086f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:14.794563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec4cd67-b7fd-464f-9490-6332f86d086f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:14.794730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec4cd67-b7fd-464f-9490-6332f86d086f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:14.815266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9ec4cd67-b7fd-464f-9490-6332f86d086f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:14.816463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9ec4cd67-b7fd-464f-9490-6332f86d086f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:14.829375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:14.879499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1067/2129] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:14.887297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af173b0c-dbe9-43ae-8f4e-26fc09ab6c1d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:14.887580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af173b0c-dbe9-43ae-8f4e-26fc09ab6c1d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:14.888423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af173b0c-dbe9-43ae-8f4e-26fc09ab6c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:14.909800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-af173b0c-dbe9-43ae-8f4e-26fc09ab6c1d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:14.911043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-af173b0c-dbe9-43ae-8f4e-26fc09ab6c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:14.919400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1063/2130] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:14.923646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec4cd67-b7fd-464f-9490-6332f86d086f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:14.933233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec4cd67-b7fd-464f-9490-6332f86d086f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:14.938519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:14.939000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:14.939282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:14.939622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:14.940292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:14.940765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:14.942512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:14.955197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1068/2131] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:14.959450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af173b0c-dbe9-43ae-8f4e-26fc09ab6c1d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:14.968603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af173b0c-dbe9-43ae-8f4e-26fc09ab6c1d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:14.973204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:14.973791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:14.974067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:14.974334 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:14.974965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:14.976918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.003067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1064/2132] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:14 2022] POST /identity/v3/projects => generated 381 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:15.007284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.017765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c210b9-3f27-403e-bd17-5f53a62428f0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.074962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.076217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.076607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.076896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.077566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.079515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.100851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1069/2133] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:15 2022] POST /identity/v3/users => generated 291 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:15.104483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.114593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1914166-cc83-4294-898d-0979c3ffa8e4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.119054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.119643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.119922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.120193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.120825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.122966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.132583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1065/2134] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:15.136308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.146792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde112a-384a-4911-9fb0-8f235e729384 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.151565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9af025-4fc9-4796-9cc7-811893b7b4ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.152153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9af025-4fc9-4796-9cc7-811893b7b4ad admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.152433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9af025-4fc9-4796-9cc7-811893b7b4ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.152708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9af025-4fc9-4796-9cc7-811893b7b4ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.153383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9af025-4fc9-4796-9cc7-811893b7b4ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.154959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9af025-4fc9-4796-9cc7-811893b7b4ad admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.165315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1070/2135] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:15.169386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.179367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f9e23a-ba16-42e7-8ed9-e74a1220c318 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.184203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.185492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.185776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.186041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] PATH_INFO: `/v3/projects/3a08b7b360bd495983479cb178841379/users/02f53a10b98b413da812c616864bea2d/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.204294 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a08b7b360bd495983479cb178841379, user_id=02f53a10b98b413da812c616864bea2d, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.207030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.221269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1066/2136] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:15 2022] PUT /identity/v3/projects/3a08b7b360bd495983479cb178841379/users/02f53a10b98b413da812c616864bea2d/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:15.234054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8e765-643c-41a6-99fa-e88902bed177 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.234505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8e765-643c-41a6-99fa-e88902bed177 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.234607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8e765-643c-41a6-99fa-e88902bed177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.266894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-63b8e765-643c-41a6-99fa-e88902bed177 None None] MFA Rules not processed for user `02f53a10b98b413da812c616864bea2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:15.268179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-63b8e765-643c-41a6-99fa-e88902bed177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:15.349483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1071/2137] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:15.353767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.439033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5033f4a-6578-49f2-8e0b-96b6f3484133 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.455658 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.456535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.457067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.457723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.458564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.460851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.473221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:15.495835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1067/2138] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:15 2022] POST /identity/v3/projects => generated 381 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:15.496182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b8e765-643c-41a6-99fa-e88902bed177 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.512067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b8e765-643c-41a6-99fa-e88902bed177 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.529023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:15.599985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.600611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.600894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.601234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.601866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.604040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.635998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1072/2139] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:15 2022] POST /identity/v3/users => generated 297 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:15.639455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.650219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c10088f-4941-439c-8cf8-c2566ce6c00b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.656223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.656818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.657132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.657404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.658044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.659644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.670418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1068/2140] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:15.675195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.688524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57709a7b-8c59-4352-88aa-0f9bd073f0b0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.693976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.694686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.694985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.695289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] PATH_INFO: `/v3/projects/45c000c11cec46c38cbc510408a823af/users/b676dfec27b9400ab01d5a9ac7bd4a36/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.712350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] RBAC: Authorizing `identity:create_grant(project_id=45c000c11cec46c38cbc510408a823af, user_id=b676dfec27b9400ab01d5a9ac7bd4a36, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.717407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.732350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1073/2141] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:15 2022] PUT /identity/v3/projects/45c000c11cec46c38cbc510408a823af/users/b676dfec27b9400ab01d5a9ac7bd4a36/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:15.736499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.746544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e476d-763d-4915-b6ca-1ab20f4a8c17 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.810182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f32dd7a-8b67-428d-ae5f-42c7c4ce6a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.810814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f32dd7a-8b67-428d-ae5f-42c7c4ce6a45 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.811090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f32dd7a-8b67-428d-ae5f-42c7c4ce6a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.811361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f32dd7a-8b67-428d-ae5f-42c7c4ce6a45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.811999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f32dd7a-8b67-428d-ae5f-42c7c4ce6a45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.813615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f32dd7a-8b67-428d-ae5f-42c7c4ce6a45 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.823188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1069/2142] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:15 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:15.829391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.839576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafb725c-09c6-4841-a863-6b2af21b504a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:15.844350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:15.845029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.845310 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.845582 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] PATH_INFO: `/v3/projects/45c000c11cec46c38cbc510408a823af/users/b676dfec27b9400ab01d5a9ac7bd4a36/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.847491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=45c000c11cec46c38cbc510408a823af, user_id=b676dfec27b9400ab01d5a9ac7bd4a36, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:15.850161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:15.860761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1074/2143] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:15 2022] PUT /identity/v3/projects/45c000c11cec46c38cbc510408a823af/users/b676dfec27b9400ab01d5a9ac7bd4a36/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:15.872832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad415f-6902-4477-8c47-b5ff8d7a322d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:15.873221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad415f-6902-4477-8c47-b5ff8d7a322d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:15.873962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad415f-6902-4477-8c47-b5ff8d7a322d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:15.903302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-89ad415f-6902-4477-8c47-b5ff8d7a322d None None] MFA Rules not processed for user `b676dfec27b9400ab01d5a9ac7bd4a36`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:15.904643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-89ad415f-6902-4477-8c47-b5ff8d7a322d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:15.979787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1070/2144] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:15 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:15.993480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:15.995520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:16.058965 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdea16d-ba22-4434-b39e-bbd68e6f7b15 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:16.062888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:16.063491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:16.063768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:16.064062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:16.073680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:16.075687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:16.085527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1075/2145] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:15 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:16.174789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad415f-6902-4477-8c47-b5ff8d7a322d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:16.185241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad415f-6902-4477-8c47-b5ff8d7a322d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:16.189216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90e72f6-8fc0-4463-a7dc-f39ed26f58ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:16.189828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e72f6-8fc0-4463-a7dc-f39ed26f58ec service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:16.190109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e72f6-8fc0-4463-a7dc-f39ed26f58ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:16.190385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90e72f6-8fc0-4463-a7dc-f39ed26f58ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:16.251048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90e72f6-8fc0-4463-a7dc-f39ed26f58ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:16.253016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90e72f6-8fc0-4463-a7dc-f39ed26f58ec service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:16.258198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:16.263729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1071/2146] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:16 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:16.810668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:16.813184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:16.888677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e56362-82ea-44ff-84cb-63b45b0f7e03 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:16.893203 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:16.893829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:16.894105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:16.894374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] PATH_INFO: `/v3/users/5bba5a1670bb4bedab9345546f000161` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:16.895554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bba5a1670bb4bedab9345546f000161)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:16.897474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:16.908200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:16.950863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bba5a1670bb4bedab9345546f000161'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:16.958279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bba5a1670bb4bedab9345546f000161'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:16.959058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bba5a1670bb4bedab9345546f000161'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:16.959705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bba5a1670bb4bedab9345546f000161'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:16.969546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bba5a1670bb4bedab9345546f000161'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:16.987807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1076/2147] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:16 2022] DELETE /identity/v3/users/5bba5a1670bb4bedab9345546f000161 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:16.992406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:16.993088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:16.993272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:17.013836 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:17.015033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:17.074841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1072/2148] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:17.085043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:17.086418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:17.146208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad342d81-33ef-4398-8f32-aef92722ca9d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:17.150897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:17.151517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:17.151950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:17.152193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:17.164407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:17.165937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:17.177979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1077/2149] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:17.468599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:17.470298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:17.491701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cf535e-e29a-415e-a1bd-2c1f0bb5e74a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:17.514161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:17.514808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:17.515086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:17.515352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] PATH_INFO: `/v3/projects/50fb7200a62b4c0a9b0b554c1dfbaf31` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:17.517775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50fb7200a62b4c0a9b0b554c1dfbaf31)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:17.519761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:17.569908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50fb7200a62b4c0a9b0b554c1dfbaf31', 'request_id': 'req-3f8770ef-202d-43fc-a52a-00068bafa9b2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.576632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50fb7200a62b4c0a9b0b554c1dfbaf31', 'request_id': 'req-3f8770ef-202d-43fc-a52a-00068bafa9b2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.584757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50fb7200a62b4c0a9b0b554c1dfbaf31', 'request_id': 'req-3f8770ef-202d-43fc-a52a-00068bafa9b2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.597212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] The token cache is being invalidate because project 50fb7200a62b4c0a9b0b554c1dfbaf31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:17.597505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.599260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.600158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1073/2150] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:17 2022] DELETE /identity/v3/projects/50fb7200a62b4c0a9b0b554c1dfbaf31 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:17.605275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:17.618853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b19072-0681-4e07-9c79-d5722c33fb68 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:17.711280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:17.712151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:17.712472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:17.712758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] PATH_INFO: `/v3/users/af5a4ee5fc224075af4e6d7d1bf98cbe` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:17.713964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af5a4ee5fc224075af4e6d7d1bf98cbe)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:17.716140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:17.757890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:17.779046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af5a4ee5fc224075af4e6d7d1bf98cbe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.792833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af5a4ee5fc224075af4e6d7d1bf98cbe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.794950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af5a4ee5fc224075af4e6d7d1bf98cbe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.795737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af5a4ee5fc224075af4e6d7d1bf98cbe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.815951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af5a4ee5fc224075af4e6d7d1bf98cbe'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:17.846158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1078/2151] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:17 2022] DELETE /identity/v3/users/af5a4ee5fc224075af4e6d7d1bf98cbe => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:18.138677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.140714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:18.153490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8770ef-202d-43fc-a52a-00068bafa9b2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.187437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.189750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:18.226086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.226832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.227339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.227818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] PATH_INFO: `/v3/users/7010544b41374ddd8c8c322fc2a80ec8` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.229194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] RBAC: Authorizing `identity:delete_user(user_id=7010544b41374ddd8c8c322fc2a80ec8)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.231204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.263991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:18.273699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0770be-72f6-4160-acfa-ecdcda501938 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.279893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.280794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.281223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.281499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] PATH_INFO: `/v3/projects/455c549470dc4403bd9476a4443d0370` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.282616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=455c549470dc4403bd9476a4443d0370)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.284562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.287215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7010544b41374ddd8c8c322fc2a80ec8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.287865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7010544b41374ddd8c8c322fc2a80ec8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.297792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7010544b41374ddd8c8c322fc2a80ec8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.299073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7010544b41374ddd8c8c322fc2a80ec8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.306927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23cc5a20-080c-40fb-b5e6-18ddbdd2c18c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7010544b41374ddd8c8c322fc2a80ec8'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.327985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1074/2152] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:18 2022] DELETE /identity/v3/users/7010544b41374ddd8c8c322fc2a80ec8 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:18.332382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.332742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.333119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.339585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '455c549470dc4403bd9476a4443d0370', 'request_id': 'req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.347278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '455c549470dc4403bd9476a4443d0370', 'request_id': 'req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.354128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '455c549470dc4403bd9476a4443d0370', 'request_id': 'req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.359126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:18.360732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:18.366258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] The token cache is being invalidate because project 455c549470dc4403bd9476a4443d0370 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:18.366557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.367707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-7e8958b6-9f4d-408c-b9ab-56ba46b96d93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:18.373118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1079/2153] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:18 2022] DELETE /identity/v3/projects/455c549470dc4403bd9476a4443d0370 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:18.374501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d848e5-7f30-45aa-a373-a0218aa7c849 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.375102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d848e5-7f30-45aa-a373-a0218aa7c849 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.375310 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d848e5-7f30-45aa-a373-a0218aa7c849 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.396842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-35d848e5-7f30-45aa-a373-a0218aa7c849 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:18.398096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-35d848e5-7f30-45aa-a373-a0218aa7c849 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:18.431895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:18.447132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1075/2154] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:18.457155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.458680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:18.466493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1080/2155] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:18.474450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3205f2af-c204-4c1d-b193-9771c6d1a346 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.474901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3205f2af-c204-4c1d-b193-9771c6d1a346 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.475178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3205f2af-c204-4c1d-b193-9771c6d1a346 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.499432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3205f2af-c204-4c1d-b193-9771c6d1a346 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:18.500921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3205f2af-c204-4c1d-b193-9771c6d1a346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:18.509846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1081/2156] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:18.514525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3205f2af-c204-4c1d-b193-9771c6d1a346 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.525087 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3205f2af-c204-4c1d-b193-9771c6d1a346 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.537805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.538437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.538789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.539131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.539854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:18.540422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.542206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.546957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420836ef-3d56-465e-987b-6e274044bbc8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.552197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.552899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.553261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.553596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.562336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1082/2157] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:19:18.566434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.570770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.572508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.577789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637ae705-d0de-4d3f-a309-dbaa2018858d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.583921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.584507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.584796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.585006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1076/2158] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:19:18.585101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.585725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.587624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.618459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1083/2159] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:18 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:18.622864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.634232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1763a54-91f3-4d04-ba19-85c256a44323 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.690092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:18.708157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.709046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.709414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.709758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.710473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.712517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.741734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1077/2160] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:18 2022] POST /identity/v3/users => generated 283 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:18.745978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.758255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c1d04f-d694-4034-a302-955533d78cd9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.763038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.763674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.763938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.764212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.764882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.766489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.777640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1084/2161] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:18.782505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.795323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b3f2d-56f1-4dd5-9555-fc967fa128f8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.801215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c3383-be19-475e-9496-a3a4d09b898b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.802163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c3383-be19-475e-9496-a3a4d09b898b admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.802445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c3383-be19-475e-9496-a3a4d09b898b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.802771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c3383-be19-475e-9496-a3a4d09b898b admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.803897 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9c3383-be19-475e-9496-a3a4d09b898b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.805626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9c3383-be19-475e-9496-a3a4d09b898b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.818425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1078/2162] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:18 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:18.822601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.833444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e29c568-1837-4fde-92b5-dea0a8ff0dd6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:18.839684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf232398-2028-464b-a015-d374de86435a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:18.840358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf232398-2028-464b-a015-d374de86435a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.840667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf232398-2028-464b-a015-d374de86435a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.841599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf232398-2028-464b-a015-d374de86435a admin admin] PATH_INFO: `/v3/projects/237b72177e0843af93c11bca27c45de5/users/e3893365df0e4a55ac9b9d70a0ebf1fc/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.859442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf232398-2028-464b-a015-d374de86435a admin admin] RBAC: Authorizing `identity:create_grant(project_id=237b72177e0843af93c11bca27c45de5, user_id=e3893365df0e4a55ac9b9d70a0ebf1fc, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:18.862469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf232398-2028-464b-a015-d374de86435a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:18.882352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1085/2163] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:18 2022] PUT /identity/v3/projects/237b72177e0843af93c11bca27c45de5/users/e3893365df0e4a55ac9b9d70a0ebf1fc/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:18.903704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51758d54-61f2-42d9-8c6d-bfd978fd352e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:18.904096 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51758d54-61f2-42d9-8c6d-bfd978fd352e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:18.904470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51758d54-61f2-42d9-8c6d-bfd978fd352e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:18.923482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf232398-2028-464b-a015-d374de86435a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:18.928866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bf232398-2028-464b-a015-d374de86435a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:18.962612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-51758d54-61f2-42d9-8c6d-bfd978fd352e None None] MFA Rules not processed for user `e3893365df0e4a55ac9b9d70a0ebf1fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:18.963953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-51758d54-61f2-42d9-8c6d-bfd978fd352e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:19.057361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf232398-2028-464b-a015-d374de86435a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.061825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1079/2164] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:18 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:19.066153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51758d54-61f2-42d9-8c6d-bfd978fd352e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.067838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.068491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.068774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.069091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] PATH_INFO: `/v3/projects/bb754458f9dd4e2494de1f45973ab341` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.070229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb754458f9dd4e2494de1f45973ab341)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.072152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.080815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51758d54-61f2-42d9-8c6d-bfd978fd352e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.087953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.088819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.089142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.089469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.090239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.092890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.132115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1080/2165] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/projects => generated 367 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:19.133579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.145743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb754458f9dd4e2494de1f45973ab341', 'request_id': 'req-4a5f222b-b31d-46ad-a289-cebc5cd16e90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:19.156285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb754458f9dd4e2494de1f45973ab341', 'request_id': 'req-4a5f222b-b31d-46ad-a289-cebc5cd16e90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:19.167393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb754458f9dd4e2494de1f45973ab341', 'request_id': 'req-4a5f222b-b31d-46ad-a289-cebc5cd16e90'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:19.179252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:19.201198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] The token cache is being invalidate because project bb754458f9dd4e2494de1f45973ab341 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:19.202090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:19.203043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4a5f222b-b31d-46ad-a289-cebc5cd16e90 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:19.206381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1086/2166] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:18 2022] DELETE /identity/v3/projects/bb754458f9dd4e2494de1f45973ab341 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:19.215875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a8750-6a8c-4938-a7c7-647d56f3f67f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.216424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a8750-6a8c-4938-a7c7-647d56f3f67f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.216767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a8750-6a8c-4938-a7c7-647d56f3f67f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.226806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0200a-d067-47b2-a4ca-5b8a5bd8ddaf admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.235616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.237235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.237796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.238187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.239030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.241774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f06a8750-6a8c-4938-a7c7-647d56f3f67f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:19.242086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.243237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f06a8750-6a8c-4938-a7c7-647d56f3f67f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:19.258328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1087/2167] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:19.267987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb49df-385e-401d-a275-a11f56355d48 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.268393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb49df-385e-401d-a275-a11f56355d48 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.268676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82eb49df-385e-401d-a275-a11f56355d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.281202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1081/2168] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/users => generated 290 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:19.283910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.292816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:19.307413 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-82eb49df-385e-401d-a275-a11f56355d48 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:19.308626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-82eb49df-385e-401d-a275-a11f56355d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:19.315906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efa4e85-6696-4774-8f69-9616afadece1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.317504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1088/2169] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:19.322950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb49df-385e-401d-a275-a11f56355d48 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.330883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.333000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.333000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.333000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.333000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.334451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.335208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82eb49df-385e-401d-a275-a11f56355d48 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.342106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.342993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.343362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.343706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.344430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:19.344836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.346503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.347538 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1082/2170] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:19 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:19.351605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.361008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1089/2171] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:19.363340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3099f6-5e43-47e7-961e-2008308dfc61 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.364309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.368243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.369044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.369389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.369747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] PATH_INFO: `/v3/projects/aeccb79802b34dd4b6f4fc002a58dba2/users/825e818c719745feae513279de0edc0a/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.375497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8240cb-1e17-46ce-93c0-508f594355d7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.380704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.381366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.381543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:19.381642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.381910 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.382539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.385217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.402745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=aeccb79802b34dd4b6f4fc002a58dba2, user_id=825e818c719745feae513279de0edc0a, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.407905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.419881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1090/2172] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/projects => generated 375 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:19.424264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.437044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1083/2173] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:19 2022] PUT /identity/v3/projects/aeccb79802b34dd4b6f4fc002a58dba2/users/825e818c719745feae513279de0edc0a/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:19.440771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.518574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a863fe-0fce-4cf1-9bde-d47f3bf1006e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.523882 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.524551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.524907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.525281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.525391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd22ee-c243-4763-9a9c-d38a86fbcd7b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.526177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.528634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.530976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.531670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.532018 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.532355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.533104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.534747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.547987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1084/2174] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:19 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:19.552212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.557034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1091/2175] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/users => generated 288 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:19.561315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.565244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4c9916-f4e1-49e7-8040-c589ecc06d3c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.570206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a223ea8a-993a-4826-8789-9b4fa1c9c7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.570945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a223ea8a-993a-4826-8789-9b4fa1c9c7d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.571226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a223ea8a-993a-4826-8789-9b4fa1c9c7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.571622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a223ea8a-993a-4826-8789-9b4fa1c9c7d8 admin admin] PATH_INFO: `/v3/projects/aeccb79802b34dd4b6f4fc002a58dba2/users/825e818c719745feae513279de0edc0a/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.571905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eaac7-0b60-4427-b780-74fd11bc1da6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.574212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a223ea8a-993a-4826-8789-9b4fa1c9c7d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aeccb79802b34dd4b6f4fc002a58dba2, user_id=825e818c719745feae513279de0edc0a, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.577150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.577751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.577855 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a223ea8a-993a-4826-8789-9b4fa1c9c7d8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.578032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.578302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.579000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.580572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.592428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1092/2177] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:19.593019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1085/2177] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:19 2022] PUT /identity/v3/projects/aeccb79802b34dd4b6f4fc002a58dba2/users/825e818c719745feae513279de0edc0a/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:19.596422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.605035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.605691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.606118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.609580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51de9887-a662-4021-9583-3c198f21853d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.638747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] MFA Rules not processed for user `825e818c719745feae513279de0edc0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:19.640560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:19.678648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.679331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.679618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.679923 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.680580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.682503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.694139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1093/2178] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:19 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:19.698301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.709082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c2864-da09-47a5-8aa7-ade1ef7d2fb7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.713845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599eae08-736d-4b1b-a430-f7d8bf64dea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.714490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599eae08-736d-4b1b-a430-f7d8bf64dea3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.714766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599eae08-736d-4b1b-a430-f7d8bf64dea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.715035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599eae08-736d-4b1b-a430-f7d8bf64dea3 admin admin] PATH_INFO: `/v3/projects/bfbfde06ed1241d0b386a26c903946ab/users/ae164d2cebd34a8baa0f9a46e4f539b4/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.730783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1086/2179] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:19.732147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599eae08-736d-4b1b-a430-f7d8bf64dea3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfbfde06ed1241d0b386a26c903946ab, user_id=ae164d2cebd34a8baa0f9a46e4f539b4, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.735010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599eae08-736d-4b1b-a430-f7d8bf64dea3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.738562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=1,keystone:SELECT=94,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:19.743746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.745397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:19.752501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1094/2180] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:19 2022] PUT /identity/v3/projects/bfbfde06ed1241d0b386a26c903946ab/users/ae164d2cebd34a8baa0f9a46e4f539b4/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:19.764307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44dfc9d-e1d6-496c-834b-b51eeb198a73 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.764589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44dfc9d-e1d6-496c-834b-b51eeb198a73 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.764861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44dfc9d-e1d6-496c-834b-b51eeb198a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.791970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e44dfc9d-e1d6-496c-834b-b51eeb198a73 None None] MFA Rules not processed for user `ae164d2cebd34a8baa0f9a46e4f539b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:19.793251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e44dfc9d-e1d6-496c-834b-b51eeb198a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:19.805751 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e50b7c1-73eb-43d9-b5ae-19788ae47575 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.815671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:19.857995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1095/2181] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:19.861885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.861885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.861885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.861885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.862397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44dfc9d-e1d6-496c-834b-b51eeb198a73 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.923973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.925170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44dfc9d-e1d6-496c-834b-b51eeb198a73 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:19.925600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.930159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:19.931201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:19.931505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:19.931786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:19.932419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:19.935343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:19.936784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1087/2182] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:19 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:19.967580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1096/2183] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/projects => generated 377 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:19.971621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:19.982462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6332436-bf1c-4c1e-bc13-02d7bfc1576a service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.028094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.031123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:20.053847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.054410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.054688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.054955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.055590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.057531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.083676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1088/2184] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:19 2022] POST /identity/v3/users => generated 289 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:20.084773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.090059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3f9e18-eb6b-47f4-bd3e-9aa36ab7fb6c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.093881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24502be1-a8c8-4ac5-9fb4-68095db2752e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.094488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24502be1-a8c8-4ac5-9fb4-68095db2752e service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.094771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24502be1-a8c8-4ac5-9fb4-68095db2752e service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.095046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24502be1-a8c8-4ac5-9fb4-68095db2752e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.095316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fb05a1-ded3-4fce-a54f-a075ecaaefea admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.096809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-24502be1-a8c8-4ac5-9fb4-68095db2752e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:20.099845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.100426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.100706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.101002 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.101649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.103225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.114249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1089/2185] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:20.118084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.124484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:20.128716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4768d-4030-413a-9ae9-5b825b66391d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.133773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.134388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.134673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.134949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.135585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.137205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.148584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1090/2186] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:20.152820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.162874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90420804-d6d7-47b7-b461-fb9b33068857 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.163732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24502be1-a8c8-4ac5-9fb4-68095db2752e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.165201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24502be1-a8c8-4ac5-9fb4-68095db2752e service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.167560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.168190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.168475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.168816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] PATH_INFO: `/v3/projects/8ece8639cb624854b1ac7527b64fbd66/users/aefe1976209d486180f003e6214f325c/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.174996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1097/2187] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:20 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:20.185192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ece8639cb624854b1ac7527b64fbd66, user_id=aefe1976209d486180f003e6214f325c, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.187870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.201637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1091/2188] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:20 2022] PUT /identity/v3/projects/8ece8639cb624854b1ac7527b64fbd66/users/aefe1976209d486180f003e6214f325c/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:20.212104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4802c10-0b4f-43af-bcaf-5fc303cf8b5f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.212382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4802c10-0b4f-43af-bcaf-5fc303cf8b5f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.212649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4802c10-0b4f-43af-bcaf-5fc303cf8b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.240836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c4802c10-0b4f-43af-bcaf-5fc303cf8b5f None None] MFA Rules not processed for user `aefe1976209d486180f003e6214f325c`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:20.241874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c4802c10-0b4f-43af-bcaf-5fc303cf8b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:20.315029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1098/2189] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:20.320285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.327213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:20.394447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f8db23-f922-4570-98eb-89a1abbfbf89 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.400181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.401093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.401400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.401668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.402293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.404221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.441043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1092/2190] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:20 2022] POST /identity/v3/projects => generated 375 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:20.444890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4802c10-0b4f-43af-bcaf-5fc303cf8b5f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.457091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4802c10-0b4f-43af-bcaf-5fc303cf8b5f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.546938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.547557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.547840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.548146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.548824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.550842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.573166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1099/2191] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:20 2022] POST /identity/v3/users => generated 294 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:20.577614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.588148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7046780-80e0-4a1a-a76f-fd5ea57ffe17 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.593746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.594326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.594608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.594881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.595523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.597158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.608804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1093/2192] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:20.613074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.623107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9df574-0d6d-4592-a929-54a015f924f4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.627730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.628375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.628656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.628952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] PATH_INFO: `/v3/projects/ff14d000b3b24f839842a5d8fa818429/users/82c4071f373a46aeaf92b038dbbcf34e/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.644990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff14d000b3b24f839842a5d8fa818429, user_id=82c4071f373a46aeaf92b038dbbcf34e, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.647628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.662008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1100/2193] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:20 2022] PUT /identity/v3/projects/ff14d000b3b24f839842a5d8fa818429/users/82c4071f373a46aeaf92b038dbbcf34e/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:20.665869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.675603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6fea03-2b99-4cf7-bf10-f9080b1cf9dc admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.742707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d36e2-2e62-491b-81d7-2f3009292267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.743317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d36e2-2e62-491b-81d7-2f3009292267 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.743592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d36e2-2e62-491b-81d7-2f3009292267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.743867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d36e2-2e62-491b-81d7-2f3009292267 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.744570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d36e2-2e62-491b-81d7-2f3009292267 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.746228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d36e2-2e62-491b-81d7-2f3009292267 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.757301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1094/2194] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:20 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:20.761628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.772350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01070b84-a1c8-45d1-90a3-682b716caac4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.777343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.777995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.778346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.778627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] PATH_INFO: `/v3/projects/ff14d000b3b24f839842a5d8fa818429/users/82c4071f373a46aeaf92b038dbbcf34e/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.780560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff14d000b3b24f839842a5d8fa818429, user_id=82c4071f373a46aeaf92b038dbbcf34e, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:20.783230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:20.794375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1101/2195] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:20 2022] PUT /identity/v3/projects/ff14d000b3b24f839842a5d8fa818429/users/82c4071f373a46aeaf92b038dbbcf34e/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:20.804754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbde5b3-80a6-4f3f-98a4-0817b222bc41 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.805104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbde5b3-80a6-4f3f-98a4-0817b222bc41 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.805383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbde5b3-80a6-4f3f-98a4-0817b222bc41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:20.834797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9bbde5b3-80a6-4f3f-98a4-0817b222bc41 None None] MFA Rules not processed for user `82c4071f373a46aeaf92b038dbbcf34e`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:20.836029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9bbde5b3-80a6-4f3f-98a4-0817b222bc41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:20.904532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:20.911021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1095/2196] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:20 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:20.920284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:20.929014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:20.976255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189fa9c0-f963-4b19-9491-33216c37234b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:20.979987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:20.980589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:20.980869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:20.981181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:21.035738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:21.037201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:21.045928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1102/2197] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:20 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:31.010020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbde5b3-80a6-4f3f-98a4-0817b222bc41 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:31.021260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbde5b3-80a6-4f3f-98a4-0817b222bc41 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:31.025245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:31.026283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:31.026562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:31.026989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:31.041824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=45 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:31.090841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:31.092492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:31.103673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1096/2198] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:31.791637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:31.800722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794f1cb4-ff37-4dd9-80c7-ede316ea62f2 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:31.805398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7fc707-00a5-422a-a823-5e411f8027b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:31.806031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7fc707-00a5-422a-a823-5e411f8027b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:31.806326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7fc707-00a5-422a-a823-5e411f8027b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:31.806593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7fc707-00a5-422a-a823-5e411f8027b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:31.816067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7fc707-00a5-422a-a823-5e411f8027b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:31.817857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7fc707-00a5-422a-a823-5e411f8027b3 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:31.828254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1103/2199] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:31 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:32.897422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:32.898907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:32.973508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152d7fe5-1f8e-496b-b677-ecf1ce0d5d9b service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:32.985121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:32.987040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:32.987424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:32.987918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] PATH_INFO: `/v3/users/02f53a10b98b413da812c616864bea2d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:32.989376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=02f53a10b98b413da812c616864bea2d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:32.991462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:33.052633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02f53a10b98b413da812c616864bea2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.053321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02f53a10b98b413da812c616864bea2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.061097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02f53a10b98b413da812c616864bea2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.064659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02f53a10b98b413da812c616864bea2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.073570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02f53a10b98b413da812c616864bea2d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.093756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1097/2200] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:32 2022] DELETE /identity/v3/users/02f53a10b98b413da812c616864bea2d => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:33.098302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:33.098802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:33.098921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:33.143153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:33.144382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:33.211380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1104/2201] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:33.223263 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:33.224751 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:33.265841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:33.331337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9652b6a3-fdc6-41f5-9e1f-db747aa06bbd admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:33.335713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:33.336399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:33.336745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:33.337109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:33.347997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:33.349509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:33.363546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1098/2202] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:33.695293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:33.696902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:33.719706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc05c3c-ce1b-4127-8968-b999870f06e9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:33.727611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:33.728129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:33.728406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:33.728734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] PATH_INFO: `/v3/projects/3a08b7b360bd495983479cb178841379` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:33.729939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a08b7b360bd495983479cb178841379)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:33.731891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:33.785556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a08b7b360bd495983479cb178841379', 'request_id': 'req-4af10399-190c-4dd7-81ad-8b75999b0980'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.796589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a08b7b360bd495983479cb178841379', 'request_id': 'req-4af10399-190c-4dd7-81ad-8b75999b0980'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.804624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a08b7b360bd495983479cb178841379', 'request_id': 'req-4af10399-190c-4dd7-81ad-8b75999b0980'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.826559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] The token cache is being invalidate because project 3a08b7b360bd495983479cb178841379 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:33.826854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.827512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.829310 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1105/2203] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:33 2022] DELETE /identity/v3/projects/3a08b7b360bd495983479cb178841379 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:33.834295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:33.847074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6408c0b-c48e-4391-a708-8480896a475d service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:33.931601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:33.932290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:33.932572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:33.932911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] PATH_INFO: `/v3/users/b676dfec27b9400ab01d5a9ac7bd4a36` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:33.934726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b676dfec27b9400ab01d5a9ac7bd4a36)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:33.937478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:33.982580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b676dfec27b9400ab01d5a9ac7bd4a36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.983225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b676dfec27b9400ab01d5a9ac7bd4a36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.989996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b676dfec27b9400ab01d5a9ac7bd4a36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.990626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b676dfec27b9400ab01d5a9ac7bd4a36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:33.998544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-52b2215f-a255-466d-a8a2-dfc814c5d6f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b676dfec27b9400ab01d5a9ac7bd4a36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:34.016421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1099/2204] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:33 2022] DELETE /identity/v3/users/b676dfec27b9400ab01d5a9ac7bd4a36 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:34.287192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:34.288655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:34.311792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af10399-190c-4dd7-81ad-8b75999b0980 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:34.368687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:34.369411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.369699 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.369975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] PATH_INFO: `/v3/projects/45c000c11cec46c38cbc510408a823af` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.371309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=45c000c11cec46c38cbc510408a823af)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:34.373877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:34.385271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:34.430414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45c000c11cec46c38cbc510408a823af', 'request_id': 'req-57eaad68-6ad6-4380-a622-1312b08b73ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:34.439548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45c000c11cec46c38cbc510408a823af', 'request_id': 'req-57eaad68-6ad6-4380-a622-1312b08b73ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:34.446487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45c000c11cec46c38cbc510408a823af', 'request_id': 'req-57eaad68-6ad6-4380-a622-1312b08b73ef'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:34.457416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] The token cache is being invalidate because project 45c000c11cec46c38cbc510408a823af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:34.457714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:34.458367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eaad68-6ad6-4380-a622-1312b08b73ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:34.459938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1106/2205] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:34 2022] DELETE /identity/v3/projects/45c000c11cec46c38cbc510408a823af => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:34.464878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf7a74-d0d2-4bc7-9c0c-7b38516b25d6 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.466265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf7a74-d0d2-4bc7-9c0c-7b38516b25d6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.466590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebf7a74-d0d2-4bc7-9c0c-7b38516b25d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.486863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-cebf7a74-d0d2-4bc7-9c0c-7b38516b25d6 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:34.488067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cebf7a74-d0d2-4bc7-9c0c-7b38516b25d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:34.531471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:34.560451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1100/2206] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:34.568071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7716dad9-9f6b-482d-b452-ec920b26832a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.568359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7716dad9-9f6b-482d-b452-ec920b26832a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.568810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7716dad9-9f6b-482d-b452-ec920b26832a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.592950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7716dad9-9f6b-482d-b452-ec920b26832a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:34.594207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7716dad9-9f6b-482d-b452-ec920b26832a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:34.614952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1107/2207] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:34.619417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebf7a74-d0d2-4bc7-9c0c-7b38516b25d6 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:34.629236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebf7a74-d0d2-4bc7-9c0c-7b38516b25d6 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:34.634558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:34.638880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.638880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.638880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.638880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:34.638880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:34.640801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:34.653381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1101/2208] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:34.658028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7716dad9-9f6b-482d-b452-ec920b26832a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:34.667404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7716dad9-9f6b-482d-b452-ec920b26832a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:34.673194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:34.673918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.674200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.674471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.675101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:34.677005 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:34.711308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1108/2209] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:34 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:34.713600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:34.726380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fa5113-5a9d-44b1-97b1-77ffccafde41 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:34.798982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:34.799590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.799869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.800272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.800773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:34.802709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:34.827163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1102/2210] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:34 2022] POST /identity/v3/users => generated 290 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:34.828571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:34.838902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f648fe79-25c5-475d-88f3-014692e3bc6d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:34.844211 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:34.844810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.845125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.845403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.846045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:34.847697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:34.857922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1109/2211] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:34.861452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:34.871347 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4575ab6-7da1-4fcf-b643-725cbab431d8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:34.875951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5580a2-c1d1-401f-9c6c-48a52248a31c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:34.876542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5580a2-c1d1-401f-9c6c-48a52248a31c admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.876840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5580a2-c1d1-401f-9c6c-48a52248a31c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.877134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5580a2-c1d1-401f-9c6c-48a52248a31c admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.877796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5580a2-c1d1-401f-9c6c-48a52248a31c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:34.879377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5580a2-c1d1-401f-9c6c-48a52248a31c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:34.890384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1103/2212] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:34.894367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:34.905606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc23a6-2c98-40a6-90e4-7a30ae731654 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:34.910201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:34.910841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.911125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.911402 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] PATH_INFO: `/v3/projects/1dd7fd412ebd4167a218d0d30720e1b8/users/08fd52eba8fb43ebb7bfaba415cbed65/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.925971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1dd7fd412ebd4167a218d0d30720e1b8, user_id=08fd52eba8fb43ebb7bfaba415cbed65, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:34.928666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:34.941206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1110/2213] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:34 2022] PUT /identity/v3/projects/1dd7fd412ebd4167a218d0d30720e1b8/users/08fd52eba8fb43ebb7bfaba415cbed65/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:34.953129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7976f2d-58bb-4f64-9428-0a1b11f9cca9 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:34.953556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7976f2d-58bb-4f64-9428-0a1b11f9cca9 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:34.953651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7976f2d-58bb-4f64-9428-0a1b11f9cca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:34.983395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a7976f2d-58bb-4f64-9428-0a1b11f9cca9 None None] MFA Rules not processed for user `08fd52eba8fb43ebb7bfaba415cbed65`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:34.984887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a7976f2d-58bb-4f64-9428-0a1b11f9cca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:35.055531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1104/2214] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:34 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:35.060194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:35.127207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb85a24-f7eb-4813-aadf-e41030d31458 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:35.131809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:35.132411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.132689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.132985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.133609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:35.135519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:35.163052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1111/2215] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:35 2022] POST /identity/v3/projects => generated 381 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:35.167144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7976f2d-58bb-4f64-9428-0a1b11f9cca9 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:35.178213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7976f2d-58bb-4f64-9428-0a1b11f9cca9 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:35.212296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:35.237913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:35.238519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.238797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.239070 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.239703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:35.241669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:35.263807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1105/2216] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:35 2022] POST /identity/v3/users => generated 297 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:35.267866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:35.278637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb64d0b6-ff2e-48b9-afb9-968a530a95fe admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:35.283194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:35.283779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.284057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.284328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.284991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:35.286567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:35.296943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1112/2217] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:35.301234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:35.311595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4a7495-7956-4738-8a7f-92bc0fa518bc admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:35.316084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:35.316727 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.317037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.317316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] PATH_INFO: `/v3/projects/1e4b5a74a720451b983740d8ff9ba81e/users/8afd3c210cff44e5bf6b77e68fb3f1ee/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.332109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e4b5a74a720451b983740d8ff9ba81e, user_id=8afd3c210cff44e5bf6b77e68fb3f1ee, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:35.334771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:35.347384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1106/2218] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:35 2022] PUT /identity/v3/projects/1e4b5a74a720451b983740d8ff9ba81e/users/8afd3c210cff44e5bf6b77e68fb3f1ee/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:35.351403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:35.361315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f629a832-4fb3-49bd-857b-98027b007ffb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:35.368918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:35.423319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb2c942-81fa-4141-bdbf-fdfd4188cf9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:35.423929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2c942-81fa-4141-bdbf-fdfd4188cf9d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.424210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2c942-81fa-4141-bdbf-fdfd4188cf9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.424483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2c942-81fa-4141-bdbf-fdfd4188cf9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.425151 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb2c942-81fa-4141-bdbf-fdfd4188cf9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:35.426782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb2c942-81fa-4141-bdbf-fdfd4188cf9d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:35.437781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1113/2219] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:35.441907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:35.452712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc387fc-d20a-4149-9f26-c90c275b2550 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:35.457373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:35.458009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.458286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.458555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] PATH_INFO: `/v3/projects/1e4b5a74a720451b983740d8ff9ba81e/users/8afd3c210cff44e5bf6b77e68fb3f1ee/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.460430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e4b5a74a720451b983740d8ff9ba81e, user_id=8afd3c210cff44e5bf6b77e68fb3f1ee, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:35.463074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:35.473112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1107/2220] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:35 2022] PUT /identity/v3/projects/1e4b5a74a720451b983740d8ff9ba81e/users/8afd3c210cff44e5bf6b77e68fb3f1ee/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:35.483309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0d6e70-cc30-4327-980e-36ec634cc40f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.483678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0d6e70-cc30-4327-980e-36ec634cc40f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.483961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0d6e70-cc30-4327-980e-36ec634cc40f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.513670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5b0d6e70-cc30-4327-980e-36ec634cc40f None None] MFA Rules not processed for user `8afd3c210cff44e5bf6b77e68fb3f1ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:35.514884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5b0d6e70-cc30-4327-980e-36ec634cc40f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:35.583799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1114/2221] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:35 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:35.596108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:35.597816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:35.662502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8587d74-4050-4189-996c-5f9fecf78685 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:35.663104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:35.663802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:35.664081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:35.664354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:35.674220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:35.675636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:35.686426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1108/2222] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:35 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:36.019710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0d6e70-cc30-4327-980e-36ec634cc40f None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:36.029796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0d6e70-cc30-4327-980e-36ec634cc40f None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:36.034467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afd3c210cff44e5bf6b77e68fb3f1ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e4b5a74a720451b983740d8ff9ba81e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:36.035153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:36.035510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:36.035855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:36.036594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:36.036997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:36.038737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:36.055344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1115/2223] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:36.056540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:36.066241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f028e-84df-4c79-9f3a-47cd1b0ed38e service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:36.070920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afd3c210cff44e5bf6b77e68fb3f1ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e4b5a74a720451b983740d8ff9ba81e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:36.071518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:36.071803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:36.072080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:36.072742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:36.074905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:36.101716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1109/2224] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:36 2022] POST /identity/v3/projects => generated 355 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:36.106650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:36.116746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7cfba3-79eb-4891-bc03-aa5396e1386c tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:36.178024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afd3c210cff44e5bf6b77e68fb3f1ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e4b5a74a720451b983740d8ff9ba81e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:36.178638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:36.178948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:36.179229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:36.180176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:36.180384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:36.182392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:36.195940 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1116/2225] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:36.239077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:36.249860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ab0872-2d2c-4776-b0ca-7323296bb938 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:36.254437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afd3c210cff44e5bf6b77e68fb3f1ee', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e4b5a74a720451b983740d8ff9ba81e', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:36.255065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:36.255353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:36.255628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] PATH_INFO: `/v3/projects/401a202811f24ff89a1e04f9ba58c02f` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:36.256740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorizing `identity:delete_project(project_id=401a202811f24ff89a1e04f9ba58c02f)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:36.258680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:36.309010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '401a202811f24ff89a1e04f9ba58c02f', 'request_id': 'req-21bf158a-fd84-4e85-8bd1-4595c90962db'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.314811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '401a202811f24ff89a1e04f9ba58c02f', 'request_id': 'req-21bf158a-fd84-4e85-8bd1-4595c90962db'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.318516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=86,keystone:INSERT=7,keystone:DELETE=7 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:36.322983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '401a202811f24ff89a1e04f9ba58c02f', 'request_id': 'req-21bf158a-fd84-4e85-8bd1-4595c90962db'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.336359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] The token cache is being invalidate because project 401a202811f24ff89a1e04f9ba58c02f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:36.336667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.337335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-21bf158a-fd84-4e85-8bd1-4595c90962db tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.338848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1110/2226] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:36 2022] DELETE /identity/v3/projects/401a202811f24ff89a1e04f9ba58c02f => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:36.537779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:36.539658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:36.553621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:36.603303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0385eeb4-a0f2-408b-bb7d-c3fec71fead8 tempest-VolumeQuotaClassesTest-1567766525 tempest-VolumeQuotaClassesTest-1567766525-project-admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:36.657285 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:36.657912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:36.658189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:36.658461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] PATH_INFO: `/v3/users/08fd52eba8fb43ebb7bfaba415cbed65` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:36.659557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08fd52eba8fb43ebb7bfaba415cbed65)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:36.661503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:36.708201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08fd52eba8fb43ebb7bfaba415cbed65'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.715627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08fd52eba8fb43ebb7bfaba415cbed65'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.716320 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08fd52eba8fb43ebb7bfaba415cbed65'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.717034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08fd52eba8fb43ebb7bfaba415cbed65'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.724853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08fd52eba8fb43ebb7bfaba415cbed65'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:36.742037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1117/2227] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:36 2022] DELETE /identity/v3/users/08fd52eba8fb43ebb7bfaba415cbed65 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:36.747189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:36.747569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:36.747851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:36.767749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:36.768878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:36.827972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1111/2228] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:36.837776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:36.839428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:36.885756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:36.901859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370cc191-d70b-4a71-8075-9b8d60273fa7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:36.906153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:36.906783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:36.907061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:36.907332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:36.916819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:36.918417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:36.928141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1118/2229] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:37.217719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:37.218766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:37.234246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804930a6-fa5a-486f-8eeb-257cd5432031 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:37.239055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:37.239812 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:37.240201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:37.240702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] PATH_INFO: `/v3/projects/1dd7fd412ebd4167a218d0d30720e1b8` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:37.242264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1dd7fd412ebd4167a218d0d30720e1b8)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:37.244429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:37.293988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1dd7fd412ebd4167a218d0d30720e1b8', 'request_id': 'req-3207dbcd-e2db-4968-b7c4-f548a64c5376'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.300983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1dd7fd412ebd4167a218d0d30720e1b8', 'request_id': 'req-3207dbcd-e2db-4968-b7c4-f548a64c5376'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.309502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1dd7fd412ebd4167a218d0d30720e1b8', 'request_id': 'req-3207dbcd-e2db-4968-b7c4-f548a64c5376'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.321613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] The token cache is being invalidate because project 1dd7fd412ebd4167a218d0d30720e1b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:37.321902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.322558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.324405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1112/2230] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:37 2022] DELETE /identity/v3/projects/1dd7fd412ebd4167a218d0d30720e1b8 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:37.328145 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:37.339023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74561056-e3be-4b0b-bb31-28cd5e6a42ac service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:37.404249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:37.405221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:37.405504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:37.405806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] PATH_INFO: `/v3/users/8afd3c210cff44e5bf6b77e68fb3f1ee` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:37.407066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8afd3c210cff44e5bf6b77e68fb3f1ee)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:37.409087 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:37.464634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8afd3c210cff44e5bf6b77e68fb3f1ee'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.473296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8afd3c210cff44e5bf6b77e68fb3f1ee'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.474135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8afd3c210cff44e5bf6b77e68fb3f1ee'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.474688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8afd3c210cff44e5bf6b77e68fb3f1ee'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.482587 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-22b4134d-485a-4132-bdc0-0aa0ec217a00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8afd3c210cff44e5bf6b77e68fb3f1ee'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.502127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1119/2231] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:37 2022] DELETE /identity/v3/users/8afd3c210cff44e5bf6b77e68fb3f1ee => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:37.781553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:37.783282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:37.798131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3207dbcd-e2db-4968-b7c4-f548a64c5376 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:37.859467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:37.860151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:37.860467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:37.860772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] PATH_INFO: `/v3/projects/1e4b5a74a720451b983740d8ff9ba81e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:37.861938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e4b5a74a720451b983740d8ff9ba81e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:37.863816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:37.881136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:37.923014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e4b5a74a720451b983740d8ff9ba81e', 'request_id': 'req-36fc8195-87a5-47a9-9df4-c67a8d309ffc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.932491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e4b5a74a720451b983740d8ff9ba81e', 'request_id': 'req-36fc8195-87a5-47a9-9df4-c67a8d309ffc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.943323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e4b5a74a720451b983740d8ff9ba81e', 'request_id': 'req-36fc8195-87a5-47a9-9df4-c67a8d309ffc'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.967785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] The token cache is being invalidate because project 1e4b5a74a720451b983740d8ff9ba81e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:37.967785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.967785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:37.967785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1113/2232] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:37 2022] DELETE /identity/v3/projects/1e4b5a74a720451b983740d8ff9ba81e => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:37.969883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018a2d44-2927-4f14-ae9a-a2835bbf196f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:37.970690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018a2d44-2927-4f14-ae9a-a2835bbf196f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:37.970968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018a2d44-2927-4f14-ae9a-a2835bbf196f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:37.993585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-018a2d44-2927-4f14-ae9a-a2835bbf196f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:37.994900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-018a2d44-2927-4f14-ae9a-a2835bbf196f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.040084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.042056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.073729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1120/2233] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:38.082858 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.083162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.083530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.102300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:38.108266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:38.109926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.120729 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1121/2234] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:38.121947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.125199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.140073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a2c57b-573c-4567-9f8e-e26788391a8c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.145651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.146236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.146519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.146849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] PATH_INFO: `/v3/users/e3893365df0e4a55ac9b9d70a0ebf1fc` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.148043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3893365df0e4a55ac9b9d70a0ebf1fc)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.148196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fc8195-87a5-47a9-9df4-c67a8d309ffc admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.150117 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.156370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.158396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.158781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.159129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] PATH_INFO: `/v3/users/ae164d2cebd34a8baa0f9a46e4f539b4` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.160920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae164d2cebd34a8baa0f9a46e4f539b4)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.163336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.214354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3893365df0e4a55ac9b9d70a0ebf1fc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.223836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3893365df0e4a55ac9b9d70a0ebf1fc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.225024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3893365df0e4a55ac9b9d70a0ebf1fc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.226165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3893365df0e4a55ac9b9d70a0ebf1fc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.232961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae164d2cebd34a8baa0f9a46e4f539b4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.233771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae164d2cebd34a8baa0f9a46e4f539b4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.237808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3893365df0e4a55ac9b9d70a0ebf1fc'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.242560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae164d2cebd34a8baa0f9a46e4f539b4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.243400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae164d2cebd34a8baa0f9a46e4f539b4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.259735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a51d156f-d581-4a3a-83bc-6307473881ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae164d2cebd34a8baa0f9a46e4f539b4'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.267670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1122/2235] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:38 2022] DELETE /identity/v3/users/e3893365df0e4a55ac9b9d70a0ebf1fc => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:38.268579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.270142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.284703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1114/2236] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:38 2022] DELETE /identity/v3/users/ae164d2cebd34a8baa0f9a46e4f539b4 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:38.291211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c5dbb7-8e36-4919-a9cb-b20a71f3ba82 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.291506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c5dbb7-8e36-4919-a9cb-b20a71f3ba82 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.291936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c5dbb7-8e36-4919-a9cb-b20a71f3ba82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.316806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-c9c5dbb7-8e36-4919-a9cb-b20a71f3ba82 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:38.316806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c9c5dbb7-8e36-4919-a9cb-b20a71f3ba82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.337741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:38.354734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57eb0524-12f3-4ca7-930b-e39925b2989b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.382571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1115/2237] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:38.383941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.384593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.384593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.406179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:38.407374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.414979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.415445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1116/2238] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:38.415639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.415877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.416222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.416397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.416903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:38.417237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.419040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.420033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.434665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1123/2239] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:38.435753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.438132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.450171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:38.491193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4972a000-55cc-4bb4-bb5f-721f377e3ada None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.495568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.496720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.497035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.497354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.505818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8ee03c-abd2-40f9-b97d-a146b9e03f99 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.509308 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.511344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.511611 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.512688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.512983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.513272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.527185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.528903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.531282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1117/2240] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:38.532108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.533715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.539258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1124/2241] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:38.549921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb054cb-6726-4f08-bdb0-c6a27eebeb3a service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.556136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.556823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.557218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.559401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.559401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.561782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.599939 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1118/2242] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:38 2022] POST /identity/v3/projects => generated 379 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:38.605079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.615687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb8c124-5bfc-471e-8374-22900cb63757 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.706428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.707274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.707655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.707935 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.709036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.711962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.743964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1125/2243] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:38 2022] POST /identity/v3/users => generated 290 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:38.745319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.758058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de85a83-5ef7-4af3-95f4-583f24e7db59 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.764313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.764954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.765268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.765521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.766168 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.768284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.789111 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1119/2244] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:38 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:38.792830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.805092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b84a890-5f31-4f3b-b6d0-426c0b950ae8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.814515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.814515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.814515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.814515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.814869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.815911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.829351 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1126/2245] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:38 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:38.833529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.849594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5293cc53-a70d-4705-a151-d60bf3bb058a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.850653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:38.853095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.857348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f07c7e-274f-433d-9f5a-1e3682a25abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.858320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f07c7e-274f-433d-9f5a-1e3682a25abc admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.858816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f07c7e-274f-433d-9f5a-1e3682a25abc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.859286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f07c7e-274f-433d-9f5a-1e3682a25abc admin admin] PATH_INFO: `/v3/projects/e2525409ffd64be1b781d951bf7dcc5f/users/01af715dd52c4af7ad7d21143d87af6f/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.883219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f07c7e-274f-433d-9f5a-1e3682a25abc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2525409ffd64be1b781d951bf7dcc5f, user_id=01af715dd52c4af7ad7d21143d87af6f, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.883995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a3fa4-01ae-43f7-bbaa-8c156d240f7f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:38.885990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f07c7e-274f-433d-9f5a-1e3682a25abc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.889755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:38.890399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.890682 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.890961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] PATH_INFO: `/v3/projects/bfbfde06ed1241d0b386a26c903946ab` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.892538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfbfde06ed1241d0b386a26c903946ab)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:38.895236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:38.906862 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1120/2246] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:38 2022] PUT /identity/v3/projects/e2525409ffd64be1b781d951bf7dcc5f/users/01af715dd52c4af7ad7d21143d87af6f/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:38.918138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:38.918506 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:38.918695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:38.949545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] MFA Rules not processed for user `01af715dd52c4af7ad7d21143d87af6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:38.950753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:38.959178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfbfde06ed1241d0b386a26c903946ab', 'request_id': 'req-83636050-f0ce-4c6f-9c48-a8a05878c9fb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.968816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfbfde06ed1241d0b386a26c903946ab', 'request_id': 'req-83636050-f0ce-4c6f-9c48-a8a05878c9fb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.978137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfbfde06ed1241d0b386a26c903946ab', 'request_id': 'req-83636050-f0ce-4c6f-9c48-a8a05878c9fb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.991024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] The token cache is being invalidate because project bfbfde06ed1241d0b386a26c903946ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:38.991324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.992008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:38.997028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1127/2247] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:38 2022] DELETE /identity/v3/projects/bfbfde06ed1241d0b386a26c903946ab => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:38.997028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.003690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.021790 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:39.026211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:39.044394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1121/2248] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:38 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:39.045303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.047260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.084525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83636050-f0ce-4c6f-9c48-a8a05878c9fb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.091060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.091358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.091643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.091921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] PATH_INFO: `/v3/projects/237b72177e0843af93c11bca27c45de5` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.093681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=237b72177e0843af93c11bca27c45de5)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.095829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.137530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea22af7-c0df-4acd-8cdb-75edf5dc1951 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.138922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.139553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.139835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.140106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] PATH_INFO: `/v3/users/aefe1976209d486180f003e6214f325c` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.141264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aefe1976209d486180f003e6214f325c)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.143363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.159621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '237b72177e0843af93c11bca27c45de5', 'request_id': 'req-e672d6a3-5791-4603-83b8-8342326e97b2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.167222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '237b72177e0843af93c11bca27c45de5', 'request_id': 'req-e672d6a3-5791-4603-83b8-8342326e97b2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.174549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '237b72177e0843af93c11bca27c45de5', 'request_id': 'req-e672d6a3-5791-4603-83b8-8342326e97b2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.187313 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] The token cache is being invalidate because project 237b72177e0843af93c11bca27c45de5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:39.187690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.188501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.190197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1128/2249] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:38 2022] DELETE /identity/v3/projects/237b72177e0843af93c11bca27c45de5 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:39.191018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.194255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.195760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aefe1976209d486180f003e6214f325c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.196490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aefe1976209d486180f003e6214f325c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.206513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aefe1976209d486180f003e6214f325c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.207220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aefe1976209d486180f003e6214f325c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.214951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aefe1976209d486180f003e6214f325c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.233593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1122/2250] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:39 2022] DELETE /identity/v3/users/aefe1976209d486180f003e6214f325c => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:39.234562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.235893 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.267274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e672d6a3-5791-4603-83b8-8342326e97b2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.334640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d663dda9-4f30-492a-8d4a-9371108b77e3 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.338162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:39.339549 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.340448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.340736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.341044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] PATH_INFO: `/v3/users/825e818c719745feae513279de0edc0a` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.342232 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=825e818c719745feae513279de0edc0a)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.344203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.365240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.365697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.365954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.366228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.366870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.368798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.375402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:39.411589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1129/2251] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:39 2022] POST /identity/v3/projects => generated 381 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:39.415969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.418884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.434104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '825e818c719745feae513279de0edc0a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.434888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '825e818c719745feae513279de0edc0a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.446526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '825e818c719745feae513279de0edc0a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.447325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '825e818c719745feae513279de0edc0a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.458797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '825e818c719745feae513279de0edc0a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.488008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1123/2252] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:39 2022] DELETE /identity/v3/users/825e818c719745feae513279de0edc0a => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:39.528051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32cf1b5-531e-4618-aa6f-4be773f83bc1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.614028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.614028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.614028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.614028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.614325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.617863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.618416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.620317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.650239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d60d07-6ebd-4ba1-9905-52d2f1103e4a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.664448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1130/2253] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:39 2022] POST /identity/v3/users => generated 297 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:39.665376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.668488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.669344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.669690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.669903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] PATH_INFO: `/v3/projects/8ece8639cb624854b1ac7527b64fbd66` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.672015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ece8639cb624854b1ac7527b64fbd66)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.674080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.675280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.690526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d3efd8-4e29-4c2f-8b2b-8969a68726c9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.697419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.698777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.699136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.702017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.702689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.704640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.724361 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1131/2254] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:39 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:39.725955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.740036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddf9f69-90cb-4749-aabb-71baa804d94f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.762433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ece8639cb624854b1ac7527b64fbd66', 'request_id': 'req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.770282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ece8639cb624854b1ac7527b64fbd66', 'request_id': 'req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.794113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ece8639cb624854b1ac7527b64fbd66', 'request_id': 'req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.822382 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] The token cache is being invalidate because project 8ece8639cb624854b1ac7527b64fbd66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:39.822985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.823859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:39.825648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1124/2255] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:39 2022] DELETE /identity/v3/projects/8ece8639cb624854b1ac7527b64fbd66 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:39.829880 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.833728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.833931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:39.834830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.835116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.835966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] PATH_INFO: `/v3/projects/eb1f7b055a034b05b622bdac99afa61d/users/ac49133df34d46cfa445c9c7d4a47a40/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.847428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:39.859233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d530ce28-11a0-48a9-b9fe-05e1eca1b29f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:39.873836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb1f7b055a034b05b622bdac99afa61d, user_id=ac49133df34d46cfa445c9c7d4a47a40, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.878491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.890996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:39.891653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:39.891984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:39.892209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] PATH_INFO: `/v3/users/82c4071f373a46aeaf92b038dbbcf34e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:39.896024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82c4071f373a46aeaf92b038dbbcf34e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:39.896979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1132/2256] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:39 2022] PUT /identity/v3/projects/eb1f7b055a034b05b622bdac99afa61d/users/ac49133df34d46cfa445c9c7d4a47a40/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:39.897928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:39.898123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:39.899823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.014601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82c4071f373a46aeaf92b038dbbcf34e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.016632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82c4071f373a46aeaf92b038dbbcf34e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.034121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82c4071f373a46aeaf92b038dbbcf34e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.035781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82c4071f373a46aeaf92b038dbbcf34e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.038962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5fd639-ad66-47a0-99d3-fa92af3df5cc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.046605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.047649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.048558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.049253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] PATH_INFO: `/v3/projects/aeccb79802b34dd4b6f4fc002a58dba2` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.050928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aeccb79802b34dd4b6f4fc002a58dba2)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.053836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.054030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82c4071f373a46aeaf92b038dbbcf34e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.063487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=72,keystone:DELETE=25,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:40.084786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1125/2257] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:39 2022] DELETE /identity/v3/users/82c4071f373a46aeaf92b038dbbcf34e => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:40.086912 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.088559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.123357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aeccb79802b34dd4b6f4fc002a58dba2', 'request_id': 'req-6d61133b-a688-47d4-9ac8-50765f9fd190'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.131889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aeccb79802b34dd4b6f4fc002a58dba2', 'request_id': 'req-6d61133b-a688-47d4-9ac8-50765f9fd190'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.141776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aeccb79802b34dd4b6f4fc002a58dba2', 'request_id': 'req-6d61133b-a688-47d4-9ac8-50765f9fd190'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.153932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] The token cache is being invalidate because project aeccb79802b34dd4b6f4fc002a58dba2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:40.154232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.155035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6d61133b-a688-47d4-9ac8-50765f9fd190 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.158025 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1133/2258] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:39 2022] DELETE /identity/v3/projects/aeccb79802b34dd4b6f4fc002a58dba2 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:40.162766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb73127-acbf-4fc1-9202-e8db3f17e864 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.163067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb73127-acbf-4fc1-9202-e8db3f17e864 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.163327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb73127-acbf-4fc1-9202-e8db3f17e864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.187979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-adb73127-acbf-4fc1-9202-e8db3f17e864 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:40.189537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-adb73127-acbf-4fc1-9202-e8db3f17e864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.200798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c773c980-8a44-498b-a78f-55b7ab966312 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.264128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1134/2259] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:40.276031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276f1283-8e60-4104-8c6a-22d0d59fdc91 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.276333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276f1283-8e60-4104-8c6a-22d0d59fdc91 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.276625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276f1283-8e60-4104-8c6a-22d0d59fdc91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.279728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.280953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.280953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.281129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.282423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.284145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.304624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1126/2260] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:40 2022] GET /identity/v3/roles => generated 1420 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:40.308348 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.313394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.319142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-276f1283-8e60-4104-8c6a-22d0d59fdc91 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:40.320892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-276f1283-8e60-4104-8c6a-22d0d59fdc91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.332705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1135/2261] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:40.337537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276f1283-8e60-4104-8c6a-22d0d59fdc91 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.341516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8749906-5e34-4b52-a19b-95c9141b2414 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.351135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b53a3-447d-48a0-b5be-21b70fcc1f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.351894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b53a3-447d-48a0-b5be-21b70fcc1f56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.352239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b53a3-447d-48a0-b5be-21b70fcc1f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.352586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b53a3-447d-48a0-b5be-21b70fcc1f56 admin admin] PATH_INFO: `/v3/projects/eb1f7b055a034b05b622bdac99afa61d/users/ac49133df34d46cfa445c9c7d4a47a40/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.354141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276f1283-8e60-4104-8c6a-22d0d59fdc91 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.354984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88b53a3-447d-48a0-b5be-21b70fcc1f56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb1f7b055a034b05b622bdac99afa61d, user_id=ac49133df34d46cfa445c9c7d4a47a40, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.358939 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88b53a3-447d-48a0-b5be-21b70fcc1f56 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.361502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.362813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.363293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.363738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.364414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:40.364728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.366380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.371042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:40.379355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1127/2262] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:40 2022] PUT /identity/v3/projects/eb1f7b055a034b05b622bdac99afa61d/users/ac49133df34d46cfa445c9c7d4a47a40/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:40.393995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.394341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.394556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.407496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1136/2263] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:40.413837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.423976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3a9223-c850-4592-b668-8be98bf8f339 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.433059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] MFA Rules not processed for user `ac49133df34d46cfa445c9c7d4a47a40`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:40.435993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.520360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.520360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.520360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.520360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.520915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.520915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.537737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1128/2264] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:40.538408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.539693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.549055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:40.556797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5f0a90-92f6-44d6-8880-683a8e72308b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.560621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1137/2265] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/projects => generated 383 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:40.561626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.564652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.566367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.567001 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.567281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.567546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] PATH_INFO: `/v3/projects/ff14d000b3b24f839842a5d8fa818429` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.568659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff14d000b3b24f839842a5d8fa818429)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.570755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.618782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff14d000b3b24f839842a5d8fa818429', 'request_id': 'req-ea602007-1bef-42c3-8a2a-b87f54c48e6c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.626483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff14d000b3b24f839842a5d8fa818429', 'request_id': 'req-ea602007-1bef-42c3-8a2a-b87f54c48e6c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.628272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178acf7b-c0a4-45d2-a921-4d35ecf5e193 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.634666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff14d000b3b24f839842a5d8fa818429', 'request_id': 'req-ea602007-1bef-42c3-8a2a-b87f54c48e6c'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.646820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] The token cache is being invalidate because project ff14d000b3b24f839842a5d8fa818429 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:40.647108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.647870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:40.649721 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1129/2266] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:40 2022] DELETE /identity/v3/projects/ff14d000b3b24f839842a5d8fa818429 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:40.654008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.654008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.680327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.680967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.681640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.681640 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.683307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.731394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea602007-1bef-42c3-8a2a-b87f54c48e6c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.736342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679e3ff4-c740-489a-8255-a6d8ab36c1fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.737087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e3ff4-c740-489a-8255-a6d8ab36c1fc admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.737374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e3ff4-c740-489a-8255-a6d8ab36c1fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.737652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679e3ff4-c740-489a-8255-a6d8ab36c1fc admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.738358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e3ff4-c740-489a-8255-a6d8ab36c1fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.740513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679e3ff4-c740-489a-8255-a6d8ab36c1fc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.740639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:40.758533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.760047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.766700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1130/2267] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/users => generated 292 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:40.766700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ca24a-fe2f-4a67-8847-4b9de5433a1a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.766700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ca24a-fe2f-4a67-8847-4b9de5433a1a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.766700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ca24a-fe2f-4a67-8847-4b9de5433a1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.771529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1138/2268] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:40 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:40.772390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.783853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa94f370-2036-403d-b9fc-d63c41411183 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.787733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-472ca24a-fe2f-4a67-8847-4b9de5433a1a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:40.788954 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-472ca24a-fe2f-4a67-8847-4b9de5433a1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.789631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.790252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.790537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.790812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.791458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.793692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.799273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1131/2269] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:40.807380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1139/2270] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:40.809277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.811131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef8291-79b7-4a1b-a539-c6bf5e667079 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.812580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef8291-79b7-4a1b-a539-c6bf5e667079 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.812913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef8291-79b7-4a1b-a539-c6bf5e667079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.819851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77a4158-d230-4670-8cce-6efe7cb7ec3e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.828279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.828993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.829279 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.829552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.830202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.831845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.849332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1140/2271] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:40 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:40.850595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.850733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a9ef8291-79b7-4a1b-a539-c6bf5e667079 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:40.852383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a9ef8291-79b7-4a1b-a539-c6bf5e667079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.861776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1132/2272] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:40.867022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f107ba2-3f0a-4996-b746-158f9a0eddb3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.868237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef8291-79b7-4a1b-a539-c6bf5e667079 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.878898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f69eca-a493-4cc8-9118-cb3ea8960f95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.879986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef8291-79b7-4a1b-a539-c6bf5e667079 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.880101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f69eca-a493-4cc8-9118-cb3ea8960f95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.880280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f69eca-a493-4cc8-9118-cb3ea8960f95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.880558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f69eca-a493-4cc8-9118-cb3ea8960f95 admin admin] PATH_INFO: `/v3/projects/de88fe3cece54e05ba8a296373c45534/users/93fa25963a3846388a72a1bcd4eb78e1/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.885703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:40.886242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.886595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.886871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.887555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:40.887875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.889797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.897865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f69eca-a493-4cc8-9118-cb3ea8960f95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de88fe3cece54e05ba8a296373c45534, user_id=93fa25963a3846388a72a1bcd4eb78e1, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:40.900746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f69eca-a493-4cc8-9118-cb3ea8960f95 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:40.902908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1133/2273] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:19:40.907377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:40.916414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1141/2274] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:40 2022] PUT /identity/v3/projects/de88fe3cece54e05ba8a296373c45534/users/93fa25963a3846388a72a1bcd4eb78e1/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:40.919741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2f629a-364b-4aaf-8dc7-c5edcf52807f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:40.928115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09afaeb5-12ee-4085-a95a-312793e53c2d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:40.928400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09afaeb5-12ee-4085-a95a-312793e53c2d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:40.928675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09afaeb5-12ee-4085-a95a-312793e53c2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:40.962468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-09afaeb5-12ee-4085-a95a-312793e53c2d None None] MFA Rules not processed for user `93fa25963a3846388a72a1bcd4eb78e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:40.963733 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-09afaeb5-12ee-4085-a95a-312793e53c2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:40.975504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:41.005209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.005848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.006164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.006443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.007473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.010035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.049545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1134/2275] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/projects => generated 373 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:41.055236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.072976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1142/2276] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:40 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:41.077604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09afaeb5-12ee-4085-a95a-312793e53c2d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.097363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09afaeb5-12ee-4085-a95a-312793e53c2d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.174613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1c8414-1524-45fd-804c-a4116c216567 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.179393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.180084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.180426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.180745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.181515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.184032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.204881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.206120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.206526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.206897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.208123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.212391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.216612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1135/2277] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:41 2022] POST /identity/v3/users => generated 287 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:41.218095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.228590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6698bc-d376-47d6-8c89-95bfb2f72e67 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.234591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.235216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.235501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.237399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.237399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.240885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.252776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1136/2278] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:41.256493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1143/2279] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:41 2022] POST /identity/v3/projects => generated 383 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:41.257363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.258159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.286317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9553bc-2bea-441b-a9e9-44b4fd3f8db8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.300483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:41.370017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee8ec3f-11e9-491b-9087-a7406128d1fa admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.375355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.375968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.376249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.376527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.377214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.378953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.386300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.387221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.387511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.387791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.388552 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.390707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.399366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1144/2280] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:41 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:41.400214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.403292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:41.419192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9806d098-3c4f-4f4c-9cac-c789660634ab admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.425564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1137/2281] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:41 2022] POST /identity/v3/users => generated 298 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:41.425564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.425823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.425823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.425823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.426089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] PATH_INFO: `/v3/users/01af715dd52c4af7ad7d21143d87af6f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.427013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=01af715dd52c4af7ad7d21143d87af6f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.429236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.438335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2d8198-78c5-4ea7-aa53-125e589d1b00 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.444309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.445157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.445477 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.446196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] PATH_INFO: `/v3/projects/6452abdab41846a5915b7ea5d81b9811/users/27dc4e15e61e44c8905e713a190a45d3/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.466387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6452abdab41846a5915b7ea5d81b9811, user_id=27dc4e15e61e44c8905e713a190a45d3, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.470481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.493447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1138/2282] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:41 2022] PUT /identity/v3/projects/6452abdab41846a5915b7ea5d81b9811/users/27dc4e15e61e44c8905e713a190a45d3/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:41.497850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.538408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01af715dd52c4af7ad7d21143d87af6f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:41.546624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:41.548368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01af715dd52c4af7ad7d21143d87af6f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:41.549751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01af715dd52c4af7ad7d21143d87af6f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:41.550832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01af715dd52c4af7ad7d21143d87af6f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:41.560479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4ce00e5e-5514-47a0-b7dc-da198a5c61ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01af715dd52c4af7ad7d21143d87af6f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:41.595797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a37b59-36de-43d3-a48f-9e2404aaf1c4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.600841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280a87c-4c20-44b2-a460-8702e1547214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.601596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280a87c-4c20-44b2-a460-8702e1547214 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.601881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280a87c-4c20-44b2-a460-8702e1547214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.602155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280a87c-4c20-44b2-a460-8702e1547214 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.602805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280a87c-4c20-44b2-a460-8702e1547214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.603614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1145/2283] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:41 2022] DELETE /identity/v3/users/01af715dd52c4af7ad7d21143d87af6f => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:41.604399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280a87c-4c20-44b2-a460-8702e1547214 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.608806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.609199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.609416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.616172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1139/2284] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:41 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:41.619585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.620165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.620961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.630637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:41.631810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:41.648679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:41.650548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:41.712869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1146/2285] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:41.713782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.716364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:41.727661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1140/2286] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:41.728399 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.730956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:41.735678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a7af38-772f-4c0a-b83b-2b4b9e92dafb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.742172 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.746029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.746786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.747396 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] PATH_INFO: `/v3/projects/23257babeb88496e8c2851a6bedf64c2/users/a8ed430dfbba45c39b845d1435a5c729/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.765205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] RBAC: Authorizing `identity:create_grant(project_id=23257babeb88496e8c2851a6bedf64c2, user_id=a8ed430dfbba45c39b845d1435a5c729, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.769724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.797963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:41.803805 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1147/2287] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:41 2022] PUT /identity/v3/projects/23257babeb88496e8c2851a6bedf64c2/users/a8ed430dfbba45c39b845d1435a5c729/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:41.804678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.816403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb6a4e6-f21e-4974-9f5a-0ecb24ab570b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.875152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e831061-3b59-486b-b1e2-06b7bb11257c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.879906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.880569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.880881 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.881188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.882524 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.883236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.883617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.884012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.920020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:41.963595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.963595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.965596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:41.967125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:41.976227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1148/2288] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:41.977218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:41.977987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1141/2289] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:41.987406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7d7c0-445d-4151-b787-69ba69bda6f8 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:41.997071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762cae6f-c36d-4664-a00a-aecc157dd7a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:41.997760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762cae6f-c36d-4664-a00a-aecc157dd7a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:41.998111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762cae6f-c36d-4664-a00a-aecc157dd7a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:41.998446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762cae6f-c36d-4664-a00a-aecc157dd7a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:41.999339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762cae6f-c36d-4664-a00a-aecc157dd7a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:42.001132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762cae6f-c36d-4664-a00a-aecc157dd7a9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:42.017033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1149/2290] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:41 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:42.018371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:42.030356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca2e1b-a322-4d95-91c4-914bc8524325 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:42.036077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:42.036829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:42.037188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:42.037473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] PATH_INFO: `/v3/projects/23257babeb88496e8c2851a6bedf64c2/users/a8ed430dfbba45c39b845d1435a5c729/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:42.039516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] RBAC: Authorizing `identity:create_grant(project_id=23257babeb88496e8c2851a6bedf64c2, user_id=a8ed430dfbba45c39b845d1435a5c729, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:42.042449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:42.056744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1142/2291] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:42 2022] PUT /identity/v3/projects/23257babeb88496e8c2851a6bedf64c2/users/a8ed430dfbba45c39b845d1435a5c729/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:42.131618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:42.132500 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:42.132809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:42.170758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] MFA Rules not processed for user `a8ed430dfbba45c39b845d1435a5c729`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:42.172024 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:42.257516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1150/2292] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:42 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:42.268421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:42.270272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:42.330807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183e3d9-e89c-448d-a779-0972066fdaad admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:42.334758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:42.335411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:42.335687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:42.335958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:42.337756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:42.401207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:42.403231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:42.412374 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1143/2293] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:42.560105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:42.561820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:42.599999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:42.618776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:42.620342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:42.693167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2423126d-0cfe-49a2-b2d8-bbd3c3b70c9e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:42.702521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:42.703518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:42.704408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:42.704698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] PATH_INFO: `/v3/users/1344a7ba15f844998e2526391f283a7c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:42.706759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1344a7ba15f844998e2526391f283a7c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:42.709512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:42.788991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda3d08-89e9-460f-938e-1246f109d9e2 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:42.797412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:42.798340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:42.798612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:42.799046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] PATH_INFO: `/v3/projects/e2525409ffd64be1b781d951bf7dcc5f` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:42.800769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2525409ffd64be1b781d951bf7dcc5f)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:42.803251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:42.805869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:42.813609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1344a7ba15f844998e2526391f283a7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.822050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1344a7ba15f844998e2526391f283a7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.823766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1344a7ba15f844998e2526391f283a7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.825598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1344a7ba15f844998e2526391f283a7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.836618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4930c9a7-e6a4-4ab1-9cd5-aa4344dbb7c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1344a7ba15f844998e2526391f283a7c'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.864995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1151/2294] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:42 2022] DELETE /identity/v3/users/1344a7ba15f844998e2526391f283a7c => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:42.868315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:42.868798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:42.869167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:42.895182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:42.896452 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:42.922702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2525409ffd64be1b781d951bf7dcc5f', 'request_id': 'req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.934453 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2525409ffd64be1b781d951bf7dcc5f', 'request_id': 'req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.953574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2525409ffd64be1b781d951bf7dcc5f', 'request_id': 'req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.968559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] The token cache is being invalidate because project e2525409ffd64be1b781d951bf7dcc5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:42.969210 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.970114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:42.976357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1144/2295] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:42 2022] DELETE /identity/v3/projects/e2525409ffd64be1b781d951bf7dcc5f => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:42.977713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:42.979472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:43.001076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1152/2296] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:43.010444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:43.013924 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:43.088637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af9cdf2-c74c-4085-9d55-7247fe40f2a6 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:43.093040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:43.093781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:43.093945 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:43.094220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] PATH_INFO: `/v3/users/ac49133df34d46cfa445c9c7d4a47a40` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:43.095411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac49133df34d46cfa445c9c7d4a47a40)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:43.096478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:43.097681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:43.110319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e62309-1367-4077-8133-4831b4e91ba0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:43.116691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc86463-a0fe-4898-b19e-2fed9db6973c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:43.117566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc86463-a0fe-4898-b19e-2fed9db6973c service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:43.117907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc86463-a0fe-4898-b19e-2fed9db6973c service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:43.118270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc86463-a0fe-4898-b19e-2fed9db6973c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:43.135047 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc86463-a0fe-4898-b19e-2fed9db6973c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:43.137794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc86463-a0fe-4898-b19e-2fed9db6973c service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:43.150594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1153/2297] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:19:43.152594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:43.153189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:43.153482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:43.159463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac49133df34d46cfa445c9c7d4a47a40'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:43.161021 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac49133df34d46cfa445c9c7d4a47a40'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:43.173731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac49133df34d46cfa445c9c7d4a47a40'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:43.174540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac49133df34d46cfa445c9c7d4a47a40'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:43.184948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac49133df34d46cfa445c9c7d4a47a40'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:43.232119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] MFA Rules not processed for user `27dc4e15e61e44c8905e713a190a45d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:43.232314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1145/2298] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:42 2022] DELETE /identity/v3/users/ac49133df34d46cfa445c9c7d4a47a40 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:43.232400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:43.360476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1154/2299] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:43.376952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:43.379986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:43.415778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:43.513098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf1fc5d-935f-4f10-b6f2-daaddb754187 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:43.523852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:43.524160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:43.524447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:43.524728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:43.542285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:43.544838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:43.556234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1146/2300] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:43 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:19:43.716904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:43.718451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:43.783079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:43.784919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:43.817804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc2191a-9d44-46fa-97f9-4c0b81311f74 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:43.838160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:43.839114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:43.839615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:43.840273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] PATH_INFO: `/v3/projects/eb1f7b055a034b05b622bdac99afa61d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:43.841814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb1f7b055a034b05b622bdac99afa61d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:43.845016 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:43.911565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623a755-c6f6-4567-99af-fa5e81113003 service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:43.953719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb1f7b055a034b05b622bdac99afa61d', 'request_id': 'req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:43.976109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb1f7b055a034b05b622bdac99afa61d', 'request_id': 'req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:43.991652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb1f7b055a034b05b622bdac99afa61d', 'request_id': 'req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.012398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] The token cache is being invalidate because project eb1f7b055a034b05b622bdac99afa61d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:44.012398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.013436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d6151e4-7b25-45ae-bcd2-f1dd9248ed0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.016999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1155/2301] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:43 2022] DELETE /identity/v3/projects/eb1f7b055a034b05b622bdac99afa61d => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:44.030581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168a1f8-ef7a-418f-8ea0-4f2e7bc53c00 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.030929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168a1f8-ef7a-418f-8ea0-4f2e7bc53c00 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.031215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168a1f8-ef7a-418f-8ea0-4f2e7bc53c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.055972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.056965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.057542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.057959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] PATH_INFO: `/v3/projects/72082f52f6244af1a940068cf4c2ac0c` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.060437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72082f52f6244af1a940068cf4c2ac0c)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.064146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.065363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:44.079757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3168a1f8-ef7a-418f-8ea0-4f2e7bc53c00 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:44.081046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3168a1f8-ef7a-418f-8ea0-4f2e7bc53c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:44.101606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1156/2302] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:44.116681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e5acb-3e42-4156-b5bb-75f3f3a71b9c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.117324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e5acb-3e42-4156-b5bb-75f3f3a71b9c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.120859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e5acb-3e42-4156-b5bb-75f3f3a71b9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.154276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72082f52f6244af1a940068cf4c2ac0c', 'request_id': 'req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.158011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e22e5acb-3e42-4156-b5bb-75f3f3a71b9c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:44.159503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e22e5acb-3e42-4156-b5bb-75f3f3a71b9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:44.160564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72082f52f6244af1a940068cf4c2ac0c', 'request_id': 'req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.164903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:44.170610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72082f52f6244af1a940068cf4c2ac0c', 'request_id': 'req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.194076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] The token cache is being invalidate because project 72082f52f6244af1a940068cf4c2ac0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:44.194722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.195562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-1fd8eeb9-2bb0-44a9-a613-bc7a1aabd1e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:44.197490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1147/2303] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:43 2022] DELETE /identity/v3/projects/72082f52f6244af1a940068cf4c2ac0c => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:44.204507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a584450-4bc9-4907-a5ee-ce081a996c2a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.204805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a584450-4bc9-4907-a5ee-ce081a996c2a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.205550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a584450-4bc9-4907-a5ee-ce081a996c2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.232759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7a584450-4bc9-4907-a5ee-ce081a996c2a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:44.234776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7a584450-4bc9-4907-a5ee-ce081a996c2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:44.245473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1157/2304] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:44.249938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e5acb-3e42-4156-b5bb-75f3f3a71b9c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.259050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e5acb-3e42-4156-b5bb-75f3f3a71b9c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.264112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.264673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.264976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.265271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.265924 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:44.266232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.269980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.283548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1158/2305] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:19:44.290792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.305049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f27393-7cc9-4eeb-9e0a-147a3e4f18f9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.312179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.312959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.313257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.313531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.314468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.316515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.319182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1148/2306] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:44.327080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a88d8-587a-403c-8139-0734df3a9631 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.327597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a88d8-587a-403c-8139-0734df3a9631 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.327863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a88d8-587a-403c-8139-0734df3a9631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.351590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-294a88d8-587a-403c-8139-0734df3a9631 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:44.352783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-294a88d8-587a-403c-8139-0734df3a9631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:44.355336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1159/2307] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:44.359760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.434479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1149/2308] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:44.440567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a88d8-587a-403c-8139-0734df3a9631 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.441909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2433af68-3e1b-4853-b60c-5f0bc41f4ca4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.446889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.447698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.447826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.448049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.449067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.451298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.452060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a88d8-587a-403c-8139-0734df3a9631 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.458678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.459714 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.461314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.461775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.462951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:44.463533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.466592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.481846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1160/2309] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/users => generated 288 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:44.482462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1150/2310] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:44.487731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.488061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.503674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4113ec8-b53d-4c67-8587-1e1f2d69804f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.507092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5441a34-fd1c-496a-8003-21ff0adce7ea admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.513568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.514284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.514468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.514748 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.515786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.515921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.516529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.516874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.517241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.517455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.518101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.521631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.536091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1161/2311] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:44.538588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.554877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4be77e7-1049-4886-a306-ddb73b3ac09d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.558398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1151/2312] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/projects => generated 383 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:44.562209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.564298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.565658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.566184 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.566470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.567790 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.569695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.582223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1162/2313] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:44.587035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.597301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1a62d8-8456-4c28-b5f7-0f7245eac64e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.628603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:44.628789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:44.632578 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf164cfd-7ed4-4192-8cc2-d61630a10e2e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.639289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.639944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.640259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.640585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.641253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.643291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.660058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.661055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.661411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.661668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] PATH_INFO: `/v3/projects/83f0808727574fbeb7b8b583299446da/users/894a4a629a804e5abb2c58d5777eed3b/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.666242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1152/2314] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/users => generated 292 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:44.670961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.679307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] RBAC: Authorizing `identity:create_grant(project_id=83f0808727574fbeb7b8b583299446da, user_id=894a4a629a804e5abb2c58d5777eed3b, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.680568 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08bcec8-ee93-4cf6-93a2-5bfdb98a6fb7 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.682036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.686068 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b26cb8-002c-44f2-883b-9b6d7d7287f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.686737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b26cb8-002c-44f2-883b-9b6d7d7287f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.686997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b26cb8-002c-44f2-883b-9b6d7d7287f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.687322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b26cb8-002c-44f2-883b-9b6d7d7287f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.688088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b26cb8-002c-44f2-883b-9b6d7d7287f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.689941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b26cb8-002c-44f2-883b-9b6d7d7287f6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.696859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1163/2315] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:44 2022] PUT /identity/v3/projects/83f0808727574fbeb7b8b583299446da/users/894a4a629a804e5abb2c58d5777eed3b/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:44.702725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1153/2316] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:44.707995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.708596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53b2661-3e1f-411a-8592-77e3f0046f2f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.709133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53b2661-3e1f-411a-8592-77e3f0046f2f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.709416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53b2661-3e1f-411a-8592-77e3f0046f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.741664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f53b2661-3e1f-411a-8592-77e3f0046f2f None None] MFA Rules not processed for user `894a4a629a804e5abb2c58d5777eed3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:44.742952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f53b2661-3e1f-411a-8592-77e3f0046f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:44.801311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfd58f-6e4d-4358-9289-ab19c9f6c0db admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.807183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.807845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.808971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.809357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.810287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.812143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.823263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1164/2317] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:44.828236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.835775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1154/2318] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:44.837514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53b2661-3e1f-411a-8592-77e3f0046f2f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.840366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbac83f-c6ce-4295-b093-e043171fe506 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.845668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6df21bf-782a-446f-b6b6-3a62f5fb9b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.846274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df21bf-782a-446f-b6b6-3a62f5fb9b50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.846561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df21bf-782a-446f-b6b6-3a62f5fb9b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.846933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df21bf-782a-446f-b6b6-3a62f5fb9b50 admin admin] PATH_INFO: `/v3/projects/1d29e2e750f948e1b0ccd4117c0d347a/users/3d75f6b795064e00b20207e6cfc22cf1/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.856817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53b2661-3e1f-411a-8592-77e3f0046f2f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:44.861630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:44.862243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.862511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.862785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.863644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.865352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.873704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6df21bf-782a-446f-b6b6-3a62f5fb9b50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d29e2e750f948e1b0ccd4117c0d347a, user_id=3d75f6b795064e00b20207e6cfc22cf1, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:44.878820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6df21bf-782a-446f-b6b6-3a62f5fb9b50 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:44.897390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1165/2319] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:44 2022] PUT /identity/v3/projects/1d29e2e750f948e1b0ccd4117c0d347a/users/3d75f6b795064e00b20207e6cfc22cf1/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:44.903508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1155/2320] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/projects => generated 375 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:44.905183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:44.905622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:44.906050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:44.907858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:44.938795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:44.941583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:45.009731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ae72d-9d78-4a5c-ac22-0807f1adc2ed admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:45.015014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:45.015619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.015900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.016174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.016650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1166/2321] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:45.016823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:45.018777 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:45.028521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:45.030743 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:45.045101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1156/2322] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:44 2022] POST /identity/v3/users => generated 294 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:45.048498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:45.063028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171704e4-9f0e-4dae-994b-a951d1c82a4a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:45.068201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:45.068817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.069311 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.069428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.070023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:45.071658 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:45.082343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:45.086519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1157/2323] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:45.087765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:45.099541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2681b19-6bf1-4dc0-a9fd-f99bf2ab5f7a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:45.105193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ac0c99-2ebc-42c1-8b8d-268695879f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:45.106405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ac0c99-2ebc-42c1-8b8d-268695879f3e admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.106617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ac0c99-2ebc-42c1-8b8d-268695879f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.106894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ac0c99-2ebc-42c1-8b8d-268695879f3e admin admin] PATH_INFO: `/v3/projects/bd4877b5071741b3b666191720a8da94/users/451947a0758e43a4bdb2426ac90e4a9f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.114408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b6e560-12ea-4699-bc2d-5caa7d0ac081 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:45.118060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:45.118778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.119055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.119570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.122274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ac0c99-2ebc-42c1-8b8d-268695879f3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd4877b5071741b3b666191720a8da94, user_id=451947a0758e43a4bdb2426ac90e4a9f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:45.124908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ac0c99-2ebc-42c1-8b8d-268695879f3e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:45.128817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:45.130338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:45.132821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:45.140447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1158/2324] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:45 2022] PUT /identity/v3/projects/bd4877b5071741b3b666191720a8da94/users/451947a0758e43a4bdb2426ac90e4a9f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:45.141756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1167/2325] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:19:45.144357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:45.150124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff19d870-2e64-4dd4-84b7-498f9f5ebcc3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.150667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff19d870-2e64-4dd4-84b7-498f9f5ebcc3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.150943 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff19d870-2e64-4dd4-84b7-498f9f5ebcc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.155851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d66c5de-228f-4c5d-947d-ee91c66206fd service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:45.172004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ff19d870-2e64-4dd4-84b7-498f9f5ebcc3 None None] MFA Rules not processed for user `27dc4e15e61e44c8905e713a190a45d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:45.173511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ff19d870-2e64-4dd4-84b7-498f9f5ebcc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:45.233511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:45.234217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.234572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.234854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.235584 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:45.237750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:45.253422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1168/2326] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:45 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:45.258252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:45.267237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1159/2327] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:45.269177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced40e19-14a3-4ccd-9f32-1efaabe43162 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:45.279423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:45.279834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.279971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.280252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] PATH_INFO: `/v3/projects/bd4877b5071741b3b666191720a8da94/users/451947a0758e43a4bdb2426ac90e4a9f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.282451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd4877b5071741b3b666191720a8da94, user_id=451947a0758e43a4bdb2426ac90e4a9f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:45.285340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:45.297357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1169/2328] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:45 2022] PUT /identity/v3/projects/bd4877b5071741b3b666191720a8da94/users/451947a0758e43a4bdb2426ac90e4a9f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:45.307540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4f7d77-bd25-4be4-8914-72737384fa64 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.307655 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4f7d77-bd25-4be4-8914-72737384fa64 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.307903 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4f7d77-bd25-4be4-8914-72737384fa64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.342765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9c4f7d77-bd25-4be4-8914-72737384fa64 None None] MFA Rules not processed for user `451947a0758e43a4bdb2426ac90e4a9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:45.343974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9c4f7d77-bd25-4be4-8914-72737384fa64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:45.423694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1160/2329] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:45 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:45.435180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:45.436870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:45.497339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8f2d76-1ab4-434f-9313-496d7712490d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:45.501006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:45.501627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:45.501909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:45.502181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:45.600398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:45.601952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:45.623377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1170/2330] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:45 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:47.392837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4f7d77-bd25-4be4-8914-72737384fa64 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:47.421050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4f7d77-bd25-4be4-8914-72737384fa64 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:47.449136 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415dadaa-4a0a-49c8-b0c4-b616032267a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:47.450424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415dadaa-4a0a-49c8-b0c4-b616032267a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:47.450853 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415dadaa-4a0a-49c8-b0c4-b616032267a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:47.451286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415dadaa-4a0a-49c8-b0c4-b616032267a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:47.454711 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-415dadaa-4a0a-49c8-b0c4-b616032267a4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:47.635331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415dadaa-4a0a-49c8-b0c4-b616032267a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:47.637488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415dadaa-4a0a-49c8-b0c4-b616032267a4 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:47.663069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1161/2331] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:47 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:48.778225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:48.790724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:48.873367 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4e37c9-2ce8-4b8d-a3b4-07be2226e94b service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:48.879828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:48.880498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:48.880876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:48.881791 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] PATH_INFO: `/v3/users/894a4a629a804e5abb2c58d5777eed3b` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:48.882902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] RBAC: Authorizing `identity:delete_user(user_id=894a4a629a804e5abb2c58d5777eed3b)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:48.885007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:48.969443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '894a4a629a804e5abb2c58d5777eed3b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:48.987645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '894a4a629a804e5abb2c58d5777eed3b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:48.990379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '894a4a629a804e5abb2c58d5777eed3b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:48.992673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '894a4a629a804e5abb2c58d5777eed3b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:49.003336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '894a4a629a804e5abb2c58d5777eed3b'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:49.038474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1171/2332] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:48 2022] DELETE /identity/v3/users/894a4a629a804e5abb2c58d5777eed3b => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:49.040861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:49.041874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:49.042307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:49.068086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:49.069843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:49.091654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:49.206393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1162/2333] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:49.220092 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:49.222356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:49.340465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d89cc8-1a7b-4050-9021-2493f57de88c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:49.346282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:49.347202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:49.347528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:49.348001 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:49.363625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:49.365336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:49.377460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1172/2334] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:50.321342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:50.323427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:50.344548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86f228c-ba1d-44fc-97cd-baa2e0d45840 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:50.361159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:50.361808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:50.362091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:50.362367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] PATH_INFO: `/v3/projects/83f0808727574fbeb7b8b583299446da` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:50.367340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=83f0808727574fbeb7b8b583299446da)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:50.369293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:50.519063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83f0808727574fbeb7b8b583299446da', 'request_id': 'req-c93025c9-fc4c-4d15-a0ad-86155e09ee02'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:50.541061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83f0808727574fbeb7b8b583299446da', 'request_id': 'req-c93025c9-fc4c-4d15-a0ad-86155e09ee02'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:50.568797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83f0808727574fbeb7b8b583299446da', 'request_id': 'req-c93025c9-fc4c-4d15-a0ad-86155e09ee02'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:50.598660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] The token cache is being invalidate because project 83f0808727574fbeb7b8b583299446da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:50.598660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:50.598660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:50.600779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1163/2335] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:50 2022] DELETE /identity/v3/projects/83f0808727574fbeb7b8b583299446da => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:50.604555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:50.618049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad787a23-2bff-446e-ac3d-6e3b4225c1e8 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:50.706453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:50.797583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:50.798236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:50.798514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:50.798782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] PATH_INFO: `/v3/users/451947a0758e43a4bdb2426ac90e4a9f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:50.809874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] RBAC: Authorizing `identity:delete_user(user_id=451947a0758e43a4bdb2426ac90e4a9f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:50.812049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:51.137511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '451947a0758e43a4bdb2426ac90e4a9f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:51.151551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '451947a0758e43a4bdb2426ac90e4a9f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:51.154608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '451947a0758e43a4bdb2426ac90e4a9f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:51.156486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '451947a0758e43a4bdb2426ac90e4a9f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:51.172391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1c039146-1ef9-4b23-9f78-738b2600a318 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '451947a0758e43a4bdb2426ac90e4a9f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:51.206111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1173/2336] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:50 2022] DELETE /identity/v3/users/451947a0758e43a4bdb2426ac90e4a9f => generated 0 bytes in 599 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:51.910209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:51.913080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:51.952529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93025c9-fc4c-4d15-a0ad-86155e09ee02 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:51.994215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:51.995186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:51.996136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.072146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] MFA Rules not processed for user `3d75f6b795064e00b20207e6cfc22cf1`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:52.074378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:52.084106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.085095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.085537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.085965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] PATH_INFO: `/v3/projects/bd4877b5071741b3b666191720a8da94` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.088390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd4877b5071741b3b666191720a8da94)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.091152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.200806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:52.210970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd4877b5071741b3b666191720a8da94', 'request_id': 'req-aed53351-35b5-4811-8042-811525eeacd2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:52.230336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd4877b5071741b3b666191720a8da94', 'request_id': 'req-aed53351-35b5-4811-8042-811525eeacd2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:52.242133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd4877b5071741b3b666191720a8da94', 'request_id': 'req-aed53351-35b5-4811-8042-811525eeacd2'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:52.258142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] The token cache is being invalidate because project bd4877b5071741b3b666191720a8da94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:52.258446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:52.260679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-aed53351-35b5-4811-8042-811525eeacd2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:52.264737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1164/2337] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:51 2022] DELETE /identity/v3/projects/bd4877b5071741b3b666191720a8da94 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:52.272799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21bcf99-a210-4d93-ae4e-09ee4cf8ecdb None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.273742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21bcf99-a210-4d93-ae4e-09ee4cf8ecdb None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.275079 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21bcf99-a210-4d93-ae4e-09ee4cf8ecdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.299339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1174/2338] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:52.310666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.310854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a21bcf99-a210-4d93-ae4e-09ee4cf8ecdb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:52.313148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a21bcf99-a210-4d93-ae4e-09ee4cf8ecdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:52.313419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:52.334850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3b3f1f-7926-489a-ae27-5b3d47844751 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.392084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1165/2339] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:52.394236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:52.405034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b50eb-3dc0-4075-b5d3-f3af60e7ee72 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.405441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b50eb-3dc0-4075-b5d3-f3af60e7ee72 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.405769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2b50eb-3dc0-4075-b5d3-f3af60e7ee72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.446353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5e2b50eb-3dc0-4075-b5d3-f3af60e7ee72 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:52.448172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5e2b50eb-3dc0-4075-b5d3-f3af60e7ee72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:52.456201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.456904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.457260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.457590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.461521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1166/2340] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:52.461717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.463613 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.468106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2b50eb-3dc0-4075-b5d3-f3af60e7ee72 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.483847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2b50eb-3dc0-4075-b5d3-f3af60e7ee72 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.490863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.491771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.491981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.492301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.493489 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:52.493808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.496591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.521475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1175/2341] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:52 2022] POST /identity/v3/projects => generated 381 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:52.523470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.532886 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1167/2342] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:19:52.539029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.554358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7898fb-73b9-46aa-a7b7-cc4eceba9b94 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.645602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.646209 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.647617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.647617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.647617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.649488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.657787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b80cc09-cc4e-4a5f-bfa7-fa83f48a39d3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.662523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.663268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.663538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.663811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.664650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.666646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.701014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1168/2343] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:52 2022] POST /identity/v3/projects => generated 385 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:52.705451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.719021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1176/2344] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:52 2022] POST /identity/v3/users => generated 297 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:52.722950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.741084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d954763b-6036-4047-915e-d59e76ee26dc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.789346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:52.849463 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89c5e0c-e787-47fe-a50f-dddaa21f9304 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.864412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.866902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.867194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.867475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.868134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.876089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.935416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1169/2345] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:52 2022] POST /identity/v3/users => generated 293 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:52.942138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.943704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.944446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.944738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.945129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.946333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.949378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.965996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfcd356-02ee-4ecd-994e-3b84c02ad1f1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.976600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1177/2346] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:52 2022] GET /identity/v3/roles => generated 1420 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:52.977746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:52.989537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.989722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc701e-e9ae-43be-bfad-0dc723cb9967 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:52.991282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.991978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.992646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:52.993632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:52.995726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:52.996419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:52.997381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:52.998080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:52.998513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] PATH_INFO: `/v3/projects/0187d00cc2bb49b3812b571c884fd809/users/61e29adae78246189f15e257e9c2f98b/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.019895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=0187d00cc2bb49b3812b571c884fd809, user_id=61e29adae78246189f15e257e9c2f98b, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:53.023375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:53.029047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1170/2347] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:52 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:53.036581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:53.051484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1178/2348] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:52 2022] PUT /identity/v3/projects/0187d00cc2bb49b3812b571c884fd809/users/61e29adae78246189f15e257e9c2f98b/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:53.053603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:53.074759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1e747a-a268-4978-8ecd-0d747b0bbf91 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:53.143487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:53.210916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:53.212054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.212341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.212676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.216379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:53.219735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:53.226299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c99f48-b06f-4abe-85e7-06a8c967a2cb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:53.234704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:53.235534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.235909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.236296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.237574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:53.241776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:53.252910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1171/2349] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:53 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:53.255063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:53.268890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1179/2350] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:53 2022] GET /identity/v3/roles => generated 1420 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:53.274296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:53.279515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a989b18-856e-459b-a7a1-3b67db183f41 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:53.291497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee4e46-c353-4cf6-a29c-4c44fa5c33a9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:53.294322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099df6c9-5109-47ee-941b-4739eb03ee6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:53.295010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099df6c9-5109-47ee-941b-4739eb03ee6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.295295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099df6c9-5109-47ee-941b-4739eb03ee6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.295575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099df6c9-5109-47ee-941b-4739eb03ee6a admin admin] PATH_INFO: `/v3/projects/d763a5486b04409ea03c1b5730493ac6/users/eda83065d4494173973a74bead6c1b17/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.306311 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:53.307170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.307288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.307540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] PATH_INFO: `/v3/projects/0187d00cc2bb49b3812b571c884fd809/users/61e29adae78246189f15e257e9c2f98b/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.316842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0187d00cc2bb49b3812b571c884fd809, user_id=61e29adae78246189f15e257e9c2f98b, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:53.320224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:53.334740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099df6c9-5109-47ee-941b-4739eb03ee6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d763a5486b04409ea03c1b5730493ac6, user_id=eda83065d4494173973a74bead6c1b17, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:53.337500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099df6c9-5109-47ee-941b-4739eb03ee6a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:53.351960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1180/2351] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:53 2022] PUT /identity/v3/projects/0187d00cc2bb49b3812b571c884fd809/users/61e29adae78246189f15e257e9c2f98b/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:53.366323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:53.369770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:53.381688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1172/2352] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:53 2022] PUT /identity/v3/projects/d763a5486b04409ea03c1b5730493ac6/users/eda83065d4494173973a74bead6c1b17/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:53.391787 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67182bd0-ac85-4e36-b265-5f53cd1d2d5b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.392337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67182bd0-ac85-4e36-b265-5f53cd1d2d5b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.392766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67182bd0-ac85-4e36-b265-5f53cd1d2d5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.514422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22da5692-92fc-46f2-b54c-ae219148ae6a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:53.522058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-67182bd0-ac85-4e36-b265-5f53cd1d2d5b None None] MFA Rules not processed for user `eda83065d4494173973a74bead6c1b17`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:53.524143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-67182bd0-ac85-4e36-b265-5f53cd1d2d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:53.672978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1173/2353] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:53.674798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67182bd0-ac85-4e36-b265-5f53cd1d2d5b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:53.747532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:53.757162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:53.757806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.758082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.758350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] PATH_INFO: `/v3/users/93fa25963a3846388a72a1bcd4eb78e1` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.759807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93fa25963a3846388a72a1bcd4eb78e1)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:53.762778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:53.867010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67182bd0-ac85-4e36-b265-5f53cd1d2d5b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:53.870022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93fa25963a3846388a72a1bcd4eb78e1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:53.882599 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:53.882946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.883966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.884339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.885282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:53.888706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:53.889700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93fa25963a3846388a72a1bcd4eb78e1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:53.892152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93fa25963a3846388a72a1bcd4eb78e1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:53.893786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93fa25963a3846388a72a1bcd4eb78e1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:53.897889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:53.916459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f440d53a-6bc1-4a45-ba16-a76dd34e1279 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93fa25963a3846388a72a1bcd4eb78e1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:53.959233 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1181/2354] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:53 2022] DELETE /identity/v3/users/93fa25963a3846388a72a1bcd4eb78e1 => generated 0 bytes in 593 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:53.965057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c390e02-43d6-4e57-87fa-50157898c4c0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:53.966457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1174/2355] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:53 2022] POST /identity/v3/projects => generated 383 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:53.966929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c390e02-43d6-4e57-87fa-50157898c4c0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:53.966929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c390e02-43d6-4e57-87fa-50157898c4c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:53.970303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:53.974738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:54.015463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1c390e02-43d6-4e57-87fa-50157898c4c0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:54.019604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1c390e02-43d6-4e57-87fa-50157898c4c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:54.179098 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee522407-866b-4e78-a634-2f6341badd9f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:54.189676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:54.191572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:54.192320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:54.193031 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:54.194228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:54.198365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:54.204258 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1182/2356] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:54.234826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1175/2357] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:53 2022] POST /identity/v3/users => generated 292 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:54.238666 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c390e02-43d6-4e57-87fa-50157898c4c0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:54.250749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c390e02-43d6-4e57-87fa-50157898c4c0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:54.255071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:54.264687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:54.277850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:54.278932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:54.279328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:54.279840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:54.280589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:54.282922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:54.307543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1183/2358] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:54.309312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:54.325344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8c4cc9-c1d2-4ccb-a0b8-4a84c0a75a25 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:54.333219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:54.335399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:54.335642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:54.336174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:54.337012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:54.341071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:54.359694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1184/2359] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:54.365448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:54.404703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bcdfdc-94bd-46c3-a247-20a29763f9d5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:54.422706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3592a027-65b5-45cd-bd12-2e8da8c2eb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:54.423496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3592a027-65b5-45cd-bd12-2e8da8c2eb7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:54.424165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3592a027-65b5-45cd-bd12-2e8da8c2eb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:54.425947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3592a027-65b5-45cd-bd12-2e8da8c2eb7c admin admin] PATH_INFO: `/v3/projects/d7f5a6dc533e43d799e24f2459a7f0af/users/0cd82aafc77448faa7acf62b39cffdf6/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:54.437641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0100d3-9df6-4bd3-b083-6475f233a6c7 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:54.451154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:54.453609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:54.454854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3592a027-65b5-45cd-bd12-2e8da8c2eb7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7f5a6dc533e43d799e24f2459a7f0af, user_id=0cd82aafc77448faa7acf62b39cffdf6, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:54.455134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:54.455580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:54.458080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3592a027-65b5-45cd-bd12-2e8da8c2eb7c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:54.485195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1185/2360] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:54 2022] PUT /identity/v3/projects/d7f5a6dc533e43d799e24f2459a7f0af/users/0cd82aafc77448faa7acf62b39cffdf6/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:54.489459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:54.497266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9051cb9-177a-486d-96ee-2505bee28c52 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:54.497541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9051cb9-177a-486d-96ee-2505bee28c52 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:54.497823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9051cb9-177a-486d-96ee-2505bee28c52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:54.500448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:54.565662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1176/2361] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:54.569093 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e9051cb9-177a-486d-96ee-2505bee28c52 None None] MFA Rules not processed for user `0cd82aafc77448faa7acf62b39cffdf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:54.583022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e9051cb9-177a-486d-96ee-2505bee28c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:54.748662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1186/2362] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:54 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:54.749929 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:54.779814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92796e4a-c011-4106-b419-93d87ca93d1d service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:54.836577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:54.887654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:54.889217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:54.889823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:54.891020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:54.892125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:54.896676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:54.942371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1177/2363] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:54 2022] POST /identity/v3/projects => generated 383 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:54.946181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9051cb9-177a-486d-96ee-2505bee28c52 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:54.959796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9051cb9-177a-486d-96ee-2505bee28c52 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:54.984139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:55.102251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:55.105763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.106109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.106471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.107200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:55.111374 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:55.177332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:55.178149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1187/2364] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:54 2022] POST /identity/v3/users => generated 298 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:55.203067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e687a7-b013-4793-b55e-e0174f0dfefa admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:55.214813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:55.215670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.215983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.218120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.218955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:55.227010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:55.261914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1178/2365] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:55 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:19:55.266592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:55.287000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcc8a66-51fe-4669-b90b-99c6c265c5d8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:55.316593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:55.318470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:55.323935 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:55.324648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.325045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.325221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] PATH_INFO: `/v3/projects/662988b4ef8d429083503a93a74a7773/users/efd78a598c96455abed1b8372f579201/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.356987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=662988b4ef8d429083503a93a74a7773, user_id=efd78a598c96455abed1b8372f579201, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:55.361054 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:55.364653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3de42a8-cfb2-4ebb-844d-c7514130f659 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:55.373989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:55.374990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.381481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.381888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] PATH_INFO: `/v3/projects/de88fe3cece54e05ba8a296373c45534` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.387183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de88fe3cece54e05ba8a296373c45534)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:55.390698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1188/2366] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:55 2022] PUT /identity/v3/projects/662988b4ef8d429083503a93a74a7773/users/efd78a598c96455abed1b8372f579201/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:55.390854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:55.392509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:55.527832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de88fe3cece54e05ba8a296373c45534', 'request_id': 'req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.539730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de88fe3cece54e05ba8a296373c45534', 'request_id': 'req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.552092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de88fe3cece54e05ba8a296373c45534', 'request_id': 'req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.576203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] The token cache is being invalidate because project de88fe3cece54e05ba8a296373c45534 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:55.576775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.579789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03072412-4208-4b4a-991c-d080d179e0c8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:55.581412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.584760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1179/2367] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:55 2022] DELETE /identity/v3/projects/de88fe3cece54e05ba8a296373c45534 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:55.591330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:55.595251 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:55.747594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:55.748369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.748650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.748966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.751649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:55.754011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:55.772146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1189/2368] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:55 2022] GET /identity/v3/roles => generated 1420 bytes in 380 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:55.777816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:55.781545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:55.791848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10c3a8e-3b8b-4f14-a2bc-5b1ed37500a3 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:55.799603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:55.800911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.801606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.802217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] PATH_INFO: `/v3/users/a8ed430dfbba45c39b845d1435a5c729` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.803927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8ed430dfbba45c39b845d1435a5c729)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:55.806303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:55.813301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b5fa5-902a-4a5f-bc48-4f6406c4de9e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:55.825398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f0542-2f07-4a0c-871c-cf63e49185cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:55.826080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f0542-2f07-4a0c-871c-cf63e49185cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.826360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f0542-2f07-4a0c-871c-cf63e49185cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.826637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f0542-2f07-4a0c-871c-cf63e49185cc admin admin] PATH_INFO: `/v3/projects/662988b4ef8d429083503a93a74a7773/users/efd78a598c96455abed1b8372f579201/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.832538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f0542-2f07-4a0c-871c-cf63e49185cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=662988b4ef8d429083503a93a74a7773, user_id=efd78a598c96455abed1b8372f579201, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:55.840327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757f0542-2f07-4a0c-871c-cf63e49185cc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:55.864759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1190/2369] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:55 2022] PUT /identity/v3/projects/662988b4ef8d429083503a93a74a7773/users/efd78a598c96455abed1b8372f579201/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:55.876410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:55.877918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:55.880946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:55.883056 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:55.924595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8ed430dfbba45c39b845d1435a5c729'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.925551 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8ed430dfbba45c39b845d1435a5c729'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.936317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8ed430dfbba45c39b845d1435a5c729'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.937338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8ed430dfbba45c39b845d1435a5c729'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.947967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8ed430dfbba45c39b845d1435a5c729'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:55.978289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] MFA Rules not processed for user `efd78a598c96455abed1b8372f579201`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:55.982682 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:55.998523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:56.035870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1180/2370] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:19:55 2022] DELETE /identity/v3/users/a8ed430dfbba45c39b845d1435a5c729 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:56.187469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1191/2371] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:55 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:56.206429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:56.208080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:56.368299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4409cdd0-79bb-46ef-a847-c00a6cd3303c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:56.387336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac685d9-9566-4d07-aba0-5287ca5b1cbd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:56.388076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac685d9-9566-4d07-aba0-5287ca5b1cbd service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:56.389583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac685d9-9566-4d07-aba0-5287ca5b1cbd service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:56.389983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac685d9-9566-4d07-aba0-5287ca5b1cbd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:56.397035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6ac685d9-9566-4d07-aba0-5287ca5b1cbd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:56.636035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac685d9-9566-4d07-aba0-5287ca5b1cbd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:56.638865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac685d9-9566-4d07-aba0-5287ca5b1cbd service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:56.663363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1181/2372] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:56.750131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:56.751669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:56.950010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b90eb1-e5ce-41de-980b-3c213a0557e1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:57.159544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:57.185368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:57.210204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:57.217440 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] PATH_INFO: `/v3/projects/23257babeb88496e8c2851a6bedf64c2` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:57.223841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23257babeb88496e8c2851a6bedf64c2)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:57.228849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:57.439079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23257babeb88496e8c2851a6bedf64c2', 'request_id': 'req-953fdf1d-f9a0-4d16-b4e2-f7842de69147'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:57.442360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:57.467747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23257babeb88496e8c2851a6bedf64c2', 'request_id': 'req-953fdf1d-f9a0-4d16-b4e2-f7842de69147'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:57.478563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23257babeb88496e8c2851a6bedf64c2', 'request_id': 'req-953fdf1d-f9a0-4d16-b4e2-f7842de69147'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:57.503600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] The token cache is being invalidate because project 23257babeb88496e8c2851a6bedf64c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:19:57.503600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:57.505575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-953fdf1d-f9a0-4d16-b4e2-f7842de69147 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:19:57.511376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1192/2373] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:19:56 2022] DELETE /identity/v3/projects/23257babeb88496e8c2851a6bedf64c2 => generated 0 bytes in 761 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:19:57.521117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bbd1ac-fa9e-459f-bd9b-09ef6f5ad6a9 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:57.521695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bbd1ac-fa9e-459f-bd9b-09ef6f5ad6a9 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:57.522050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bbd1ac-fa9e-459f-bd9b-09ef6f5ad6a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:57.569052 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-46bbd1ac-fa9e-459f-bd9b-09ef6f5ad6a9 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:57.571410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-46bbd1ac-fa9e-459f-bd9b-09ef6f5ad6a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:57.696532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1182/2374] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:57.707437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708ae023-2feb-4ecb-b8f8-7fd437017158 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:57.708060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708ae023-2feb-4ecb-b8f8-7fd437017158 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:57.709327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708ae023-2feb-4ecb-b8f8-7fd437017158 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:57.750784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-708ae023-2feb-4ecb-b8f8-7fd437017158 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:57.752032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-708ae023-2feb-4ecb-b8f8-7fd437017158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:57.782592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1193/2375] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:19:57.798153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bbd1ac-fa9e-459f-bd9b-09ef6f5ad6a9 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:57.818197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bbd1ac-fa9e-459f-bd9b-09ef6f5ad6a9 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:57.828451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:57.829398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:57.830175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:57.830579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:57.832911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:19:57.833394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:57.835570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:57.851985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:57.873540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1183/2376] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:19:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:19:57.878530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708ae023-2feb-4ecb-b8f8-7fd437017158 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:57.920088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708ae023-2feb-4ecb-b8f8-7fd437017158 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:57.932106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:57.934052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:57.934857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:57.937403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:57.941801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:57.943783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:58.013080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1194/2377] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:19:57 2022] POST /identity/v3/projects => generated 377 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:19:58.013900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:58.033459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad30c6a6-3093-469b-896a-ae1ae0dd5905 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:58.178515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:58.179156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:58.179832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:58.180114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:58.180896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:58.184962 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:58.223937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1184/2378] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:19:58 2022] POST /identity/v3/users => generated 289 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:19:58.228126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:58.250867 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf56cc5-eefd-4dfe-b910-7a08395cbff5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:58.255873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:58.256644 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:58.257077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:58.257692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:58.258561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:58.260388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:58.285356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1195/2379] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:58 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:58.289762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:58.303441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2386362-73df-4305-8215-b9e43819716f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:58.335003 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae873407-90d8-4832-8ff1-0ff9bf5c0d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:58.335811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae873407-90d8-4832-8ff1-0ff9bf5c0d14 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:58.336246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae873407-90d8-4832-8ff1-0ff9bf5c0d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:58.337435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae873407-90d8-4832-8ff1-0ff9bf5c0d14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:58.338245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae873407-90d8-4832-8ff1-0ff9bf5c0d14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:58.343215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae873407-90d8-4832-8ff1-0ff9bf5c0d14 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:58.378794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:58.379643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1185/2380] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:19:58 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:19:58.395434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8922d832-1fef-41b0-b9c8-1bae1afc8a43 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:58.402453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:58.403758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:58.404356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:58.404865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] PATH_INFO: `/v3/projects/721dd52b82194e4d91113d4d688900ae/users/dea6fd8e6a374cd5b12ec7d99f285203/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:58.447684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=721dd52b82194e4d91113d4d688900ae, user_id=dea6fd8e6a374cd5b12ec7d99f285203, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:58.454529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:58.490554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1196/2381] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:19:58 2022] PUT /identity/v3/projects/721dd52b82194e4d91113d4d688900ae/users/dea6fd8e6a374cd5b12ec7d99f285203/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:19:58.491963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:58.492262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:58.492536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:58.539040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:19:58.545285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:58.706363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1186/2382] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:19:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:19:58.718238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:58.731174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:58.875315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:19:58.876809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:58.893802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684f57a8-ac40-4489-b1e2-2ebf1310fa9b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:58.902749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:58.903372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:58.903651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:58.903920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:58.922809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:58.924281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:58.941328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1197/2383] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:19:58.985043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f46d6-4a7d-4b6f-828c-d18836534e1b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:19:58.990125 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e2b31c-c9ea-419d-b324-27e95b49a855 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:19:58.990766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e2b31c-c9ea-419d-b324-27e95b49a855 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:19:58.991053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e2b31c-c9ea-419d-b324-27e95b49a855 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:19:58.991331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e2b31c-c9ea-419d-b324-27e95b49a855 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:19:58.994391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f3e2b31c-c9ea-419d-b324-27e95b49a855 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:19:59.080406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:19:59.158888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e2b31c-c9ea-419d-b324-27e95b49a855 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:19:59.160824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e2b31c-c9ea-419d-b324-27e95b49a855 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:19:59.190480 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1187/2384] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:19:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:00.533111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:00.536106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:00.564036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27d77e1-fd5d-4dc0-ba16-f8d87140c755 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:00.570034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:00.570897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:00.571275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:00.572225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:00.575218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:00.745823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:00.748451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:00.766317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1198/2385] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:00 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:01.717580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a658ee-053f-440a-bdb4-e4a84dc75c78 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:01.718405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a658ee-053f-440a-bdb4-e4a84dc75c78 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:01.718879 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a658ee-053f-440a-bdb4-e4a84dc75c78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:01.772626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a9a658ee-053f-440a-bdb4-e4a84dc75c78 None None] MFA Rules not processed for user `61e29adae78246189f15e257e9c2f98b`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:01.775433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a9a658ee-053f-440a-bdb4-e4a84dc75c78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:01.926849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1188/2386] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:01 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:01.957059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:01.959899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:01.983538 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:02.094575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73382b0e-22f1-46e7-856d-726b1dbcc55a service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:02.105780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:02.107735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:02.108788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:02.109284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:02.114680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:02.240297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:02.242223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:02.267589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1199/2387] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:02.524481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:02.525383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:02.525884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:02.575197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:02.592368 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:02.822780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1189/2388] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:20:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:20:02.915820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:02.917932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:03.079274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e700ef-9baf-4cf9-a10c-300c44cfd025 service glance] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:03.092909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff05be2f-05c7-4478-aaaa-7e7aac01a264 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:03.094530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05be2f-05c7-4478-aaaa-7e7aac01a264 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:03.095052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05be2f-05c7-4478-aaaa-7e7aac01a264 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:03.095509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05be2f-05c7-4478-aaaa-7e7aac01a264 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:03.125249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05be2f-05c7-4478-aaaa-7e7aac01a264 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:03.127251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05be2f-05c7-4478-aaaa-7e7aac01a264 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:03.139719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1200/2389] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:20:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:20:05.837188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:05.844696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:05.863802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be276544-18f6-42a9-af8b-f49cd023af53 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:05.870891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:05.871630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:05.871968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:05.872304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] PATH_INFO: `/v3/users/eda83065d4494173973a74bead6c1b17` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:05.873565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eda83065d4494173973a74bead6c1b17)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:05.876044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:06.092241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eda83065d4494173973a74bead6c1b17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:06.093950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eda83065d4494173973a74bead6c1b17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:06.105412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eda83065d4494173973a74bead6c1b17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:06.106456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eda83065d4494173973a74bead6c1b17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:06.109647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:06.124279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eda83065d4494173973a74bead6c1b17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:06.185150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1190/2390] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:20:05 2022] DELETE /identity/v3/users/eda83065d4494173973a74bead6c1b17 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:06.189971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:06.190701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:06.190807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:06.298891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:06.300190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:06.457004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:06.494340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1201/2391] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:06.506795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:06.509831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:06.675186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a94854-3074-4863-942e-f440ab192390 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:06.680679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:06.681964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:06.682896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:06.683450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:06.697497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:06.699193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:06.724028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1191/2392] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:07.326442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:07.329257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:07.352502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cd6779-f70f-4605-b926-012eaf32ebb3 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:07.372106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:07.373898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:07.375385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:07.376760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] PATH_INFO: `/v3/users/de6cb1890f484838bab3f3b662d82606` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:07.383290 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de6cb1890f484838bab3f3b662d82606)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:07.386921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:07.395486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:07.397198 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:07.450252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf4438-b374-46f2-beab-6ec8ae47c5f4 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:07.479007 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:07.482843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:07.486051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:07.486663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] PATH_INFO: `/v3/projects/d763a5486b04409ea03c1b5730493ac6` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:07.497558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d763a5486b04409ea03c1b5730493ac6)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:07.517481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:07.524236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de6cb1890f484838bab3f3b662d82606'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.555891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de6cb1890f484838bab3f3b662d82606'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.560764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de6cb1890f484838bab3f3b662d82606'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.562142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de6cb1890f484838bab3f3b662d82606'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.578362 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0e940b8b-f9bb-4a31-8bea-8f3407ffa528 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de6cb1890f484838bab3f3b662d82606'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.638933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd763a5486b04409ea03c1b5730493ac6', 'request_id': 'req-2dda24a8-8394-4639-a038-fb2efce5196a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.657166 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd763a5486b04409ea03c1b5730493ac6', 'request_id': 'req-2dda24a8-8394-4639-a038-fb2efce5196a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.663716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1202/2393] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:20:07 2022] DELETE /identity/v3/users/de6cb1890f484838bab3f3b662d82606 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:07.666711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:07.667425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:07.668835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:07.673629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd763a5486b04409ea03c1b5730493ac6', 'request_id': 'req-2dda24a8-8394-4639-a038-fb2efce5196a'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.698286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] The token cache is being invalidate because project d763a5486b04409ea03c1b5730493ac6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:20:07.698766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.699813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:07.702228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1192/2394] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:20:07 2022] DELETE /identity/v3/projects/d763a5486b04409ea03c1b5730493ac6 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:07.709998 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:07.712993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:07.730670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:07.734337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:07.914164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda24a8-8394-4639-a038-fb2efce5196a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:07.926678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=88 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:07.939720 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1203/2395] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:07.952794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:07.954833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:08.082195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:08.082858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:08.085423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:08.085734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] PATH_INFO: `/v3/users/0cd82aafc77448faa7acf62b39cffdf6` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:08.090126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cd82aafc77448faa7acf62b39cffdf6)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:08.095101 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:08.122933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d5f16a-3f75-4456-bb46-7e96346379a5 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:08.132954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0de2e84-bbf8-49cb-9cdb-ecfd9071cfcc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:08.134197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0de2e84-bbf8-49cb-9cdb-ecfd9071cfcc service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:08.134485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0de2e84-bbf8-49cb-9cdb-ecfd9071cfcc service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:08.134762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0de2e84-bbf8-49cb-9cdb-ecfd9071cfcc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:08.170633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0de2e84-bbf8-49cb-9cdb-ecfd9071cfcc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:08.175491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0de2e84-bbf8-49cb-9cdb-ecfd9071cfcc service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:08.211464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1204/2396] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:20:08.236405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cd82aafc77448faa7acf62b39cffdf6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:08.239683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cd82aafc77448faa7acf62b39cffdf6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:08.257961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cd82aafc77448faa7acf62b39cffdf6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:08.259023 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cd82aafc77448faa7acf62b39cffdf6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:08.279244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cd82aafc77448faa7acf62b39cffdf6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:08.342971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1193/2397] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:20:07 2022] DELETE /identity/v3/users/0cd82aafc77448faa7acf62b39cffdf6 => generated 0 bytes in 634 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:09.043271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29e1c4-31e3-4aa5-8f73-ff94ff06dd7c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:09.043876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29e1c4-31e3-4aa5-8f73-ff94ff06dd7c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:09.044105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29e1c4-31e3-4aa5-8f73-ff94ff06dd7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:09.153161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-6e29e1c4-31e3-4aa5-8f73-ff94ff06dd7c None None] MFA Rules not processed for user `27dc4e15e61e44c8905e713a190a45d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:09.154245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-6e29e1c4-31e3-4aa5-8f73-ff94ff06dd7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:09.202113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:09.410137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1205/2398] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 368 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:09.818397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:09.820288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:09.998272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042e0806-2f22-4509-93c3-5ef9f85c0e02 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:10.017016 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:10.017949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:10.018383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:10.018827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] PATH_INFO: `/v3/projects/f8b249a356c44057aa75318ff1ec4b1d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:10.020144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8b249a356c44057aa75318ff1ec4b1d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:10.022291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:10.109309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:10.156059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8b249a356c44057aa75318ff1ec4b1d', 'request_id': 'req-bce0bdde-618f-4cee-904f-625e362f6f07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.164179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8b249a356c44057aa75318ff1ec4b1d', 'request_id': 'req-bce0bdde-618f-4cee-904f-625e362f6f07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.178504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8b249a356c44057aa75318ff1ec4b1d', 'request_id': 'req-bce0bdde-618f-4cee-904f-625e362f6f07'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.217176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] The token cache is being invalidate because project f8b249a356c44057aa75318ff1ec4b1d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:20:10.217612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.222786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.224868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1194/2399] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:20:09 2022] DELETE /identity/v3/projects/f8b249a356c44057aa75318ff1ec4b1d => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:10.231264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca137ce8-9bc6-470c-92b0-68b1cd855461 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:10.232271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca137ce8-9bc6-470c-92b0-68b1cd855461 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:10.232771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca137ce8-9bc6-470c-92b0-68b1cd855461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:10.309219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ca137ce8-9bc6-470c-92b0-68b1cd855461 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:10.310904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ca137ce8-9bc6-470c-92b0-68b1cd855461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:10.347355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:10.350015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:10.422241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1206/2400] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:10.442191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255b93bd-70f4-4dec-ac7c-9d8d73030fcd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:10.443485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255b93bd-70f4-4dec-ac7c-9d8d73030fcd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:10.443977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255b93bd-70f4-4dec-ac7c-9d8d73030fcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:10.551190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-255b93bd-70f4-4dec-ac7c-9d8d73030fcd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:10.554941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-255b93bd-70f4-4dec-ac7c-9d8d73030fcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:10.594636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0bdde-618f-4cee-904f-625e362f6f07 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:10.596455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1207/2401] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:10.604566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255b93bd-70f4-4dec-ac7c-9d8d73030fcd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:10.616465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:10.621795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:10.623699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:10.625223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] PATH_INFO: `/v3/projects/d7f5a6dc533e43d799e24f2459a7f0af` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:10.628989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7f5a6dc533e43d799e24f2459a7f0af)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:10.636551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255b93bd-70f4-4dec-ac7c-9d8d73030fcd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:10.638195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:10.654235 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:10.654904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:10.655430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:10.655719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:10.656555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:20:10.657528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:10.659355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:10.684348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1208/2402] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:20:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:20:10.688662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:10.709631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5d494a-3ee6-46b2-93f9-513a0b984474 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:10.727605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:10.727605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:10.727605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:10.727605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:10.728178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:10.728178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:10.796006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7f5a6dc533e43d799e24f2459a7f0af', 'request_id': 'req-5faf9d78-2a98-49a3-9641-1a733b139e4f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.801939 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1209/2403] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:20:10 2022] POST /identity/v3/projects => generated 371 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:20:10.806152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:10.829650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7f5a6dc533e43d799e24f2459a7f0af', 'request_id': 'req-5faf9d78-2a98-49a3-9641-1a733b139e4f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.844710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7f5a6dc533e43d799e24f2459a7f0af', 'request_id': 'req-5faf9d78-2a98-49a3-9641-1a733b139e4f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.874859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] The token cache is being invalidate because project d7f5a6dc533e43d799e24f2459a7f0af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:20:10.874859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.875178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:10.878086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1195/2404] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:20:10 2022] DELETE /identity/v3/projects/d7f5a6dc533e43d799e24f2459a7f0af => generated 0 bytes in 532 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:10.882627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:10.884112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:10.977052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f4830-de48-478e-83a8-052e1bba44ec admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:10.983097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:10.983838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:10.984191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:10.984664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:10.985427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:10.985724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:10.988325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:11.022944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1210/2405] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:20:10 2022] POST /identity/v3/users => generated 286 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:20:11.027848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:11.029659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:11.037487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf9d78-2a98-49a3-9641-1a733b139e4f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:11.038643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:11.044550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04762602-9540-4ca2-b23d-1bad6c4b6550 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:11.046935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:11.047734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:11.048014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:11.048295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] PATH_INFO: `/v3/users/efd78a598c96455abed1b8372f579201` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:11.050006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=efd78a598c96455abed1b8372f579201)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:11.053795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:11.060132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:11.061030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:11.061415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:11.061788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:11.062545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:11.064259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:11.096738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1211/2406] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:20:11 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:20:11.099026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:11.127321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139cea08-d649-4b83-8cd9-4e3b8e0cfcf2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:11.134503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efd78a598c96455abed1b8372f579201'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:11.137208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efd78a598c96455abed1b8372f579201'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:11.139401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:11.140247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:11.140600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:11.140960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:11.141704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:11.145588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:11.175912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1212/2407] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:20:11 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:20:11.177273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:11.184328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:11.204412 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efd78a598c96455abed1b8372f579201'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:11.206460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efd78a598c96455abed1b8372f579201'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:11.215183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5540a6-22c3-4f5c-8f98-23a75d8a9f87 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:11.227378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f98a73fa-246e-4dd6-a0ca-7e01da828e38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efd78a598c96455abed1b8372f579201'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:11.228598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:11.229365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:11.229643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:11.229913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] PATH_INFO: `/v3/projects/3d7275e32e964dcd8296d442d1bf1ea1/users/d3942f001f1144e9bd79377f7b132e24/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:11.263982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d7275e32e964dcd8296d442d1bf1ea1, user_id=d3942f001f1144e9bd79377f7b132e24, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:11.269129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:11.301967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1196/2409] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:20:10 2022] DELETE /identity/v3/users/efd78a598c96455abed1b8372f579201 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:20:11.306171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1213/2409] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:20:11 2022] PUT /identity/v3/projects/3d7275e32e964dcd8296d442d1bf1ea1/users/d3942f001f1144e9bd79377f7b132e24/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:11.308594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8ca8b2-7a80-4436-a9fa-abe8a3bb56cd None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:11.309290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8ca8b2-7a80-4436-a9fa-abe8a3bb56cd None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:11.309832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8ca8b2-7a80-4436-a9fa-abe8a3bb56cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:11.363692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2e8ca8b2-7a80-4436-a9fa-abe8a3bb56cd None None] MFA Rules not processed for user `d3942f001f1144e9bd79377f7b132e24`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:11.365901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2e8ca8b2-7a80-4436-a9fa-abe8a3bb56cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:11.497746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1197/2410] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:11 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:11.515006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:11.517896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:11.781523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf978ba6-3a5d-4b0e-a610-e4a4d5fb6586 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:11.917000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac58739-c696-4597-af7b-aa14e7e07b00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:11.917636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac58739-c696-4597-af7b-aa14e7e07b00 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:11.917975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac58739-c696-4597-af7b-aa14e7e07b00 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:11.918198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac58739-c696-4597-af7b-aa14e7e07b00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:11.941983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac58739-c696-4597-af7b-aa14e7e07b00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:11.946580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac58739-c696-4597-af7b-aa14e7e07b00 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:11.965039 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1214/2411] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:11 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:12.014108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8ca8b2-7a80-4436-a9fa-abe8a3bb56cd None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:12.149547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:12.161562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8ca8b2-7a80-4436-a9fa-abe8a3bb56cd None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:12.181564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:12.182284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:12.182955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:12.182955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] PATH_INFO: `/v3/users/d3942f001f1144e9bd79377f7b132e24` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:12.197085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d3942f001f1144e9bd79377f7b132e24)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:12.199393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:12.331436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd3942f001f1144e9bd79377f7b132e24'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:12.332146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd3942f001f1144e9bd79377f7b132e24'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:12.341577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd3942f001f1144e9bd79377f7b132e24'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:12.343207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd3942f001f1144e9bd79377f7b132e24'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:12.364955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd3942f001f1144e9bd79377f7b132e24'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:12.404010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1198/2412] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:20:12 2022] DELETE /identity/v3/users/d3942f001f1144e9bd79377f7b132e24 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:20:12.409709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:12.410295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:12.410295 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:12.446906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:12.448915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:12.612436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:12.614202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:12.716122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1215/2413] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:12.728032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:12.729757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:12.781854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:12.793218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d25def-aab9-4584-84f8-ec0b36ffe4f4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:12.805008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:12.809798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:12.810386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:12.810669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] PATH_INFO: `/v3/projects/662988b4ef8d429083503a93a74a7773` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:12.814147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=662988b4ef8d429083503a93a74a7773)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:12.816110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:12.901758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d875bbb-05a2-420d-aa28-8e2e8df8a9f1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:12.911405 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83142b1-412f-4fbf-bed0-e176da72bd90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:12.912158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83142b1-412f-4fbf-bed0-e176da72bd90 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:12.912456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83142b1-412f-4fbf-bed0-e176da72bd90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:12.913519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83142b1-412f-4fbf-bed0-e176da72bd90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:12.980179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83142b1-412f-4fbf-bed0-e176da72bd90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:12.984549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83142b1-412f-4fbf-bed0-e176da72bd90 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:13.001629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '662988b4ef8d429083503a93a74a7773', 'request_id': 'req-68da21b3-5dd9-439f-9459-3dabb0b574e3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:13.020601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1216/2414] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:20:13.030678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '662988b4ef8d429083503a93a74a7773', 'request_id': 'req-68da21b3-5dd9-439f-9459-3dabb0b574e3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:13.042623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '662988b4ef8d429083503a93a74a7773', 'request_id': 'req-68da21b3-5dd9-439f-9459-3dabb0b574e3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:13.075940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] The token cache is being invalidate because project 662988b4ef8d429083503a93a74a7773 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:20:13.076278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:13.082661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-68da21b3-5dd9-439f-9459-3dabb0b574e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:13.087676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1199/2415] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:20:12 2022] DELETE /identity/v3/projects/662988b4ef8d429083503a93a74a7773 => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:13.092563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52c8c9-c303-4621-86c6-011f72eb45a2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:13.092857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52c8c9-c303-4621-86c6-011f72eb45a2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:13.093157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52c8c9-c303-4621-86c6-011f72eb45a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:13.158655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-7a52c8c9-c303-4621-86c6-011f72eb45a2 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:13.170395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7a52c8c9-c303-4621-86c6-011f72eb45a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:13.400179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1217/2416] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:13.401970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82048ef4-92b1-4abe-bc61-5b165955f78a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:13.402265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82048ef4-92b1-4abe-bc61-5b165955f78a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:13.402632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82048ef4-92b1-4abe-bc61-5b165955f78a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:13.433516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:13.482774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-82048ef4-92b1-4abe-bc61-5b165955f78a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:13.484748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-82048ef4-92b1-4abe-bc61-5b165955f78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:13.515522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1200/2417] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:13.521812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52c8c9-c303-4621-86c6-011f72eb45a2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:13.535754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52c8c9-c303-4621-86c6-011f72eb45a2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:13.542909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:13.545412 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:13.546726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:13.547224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:13.549445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:20:13.550044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:13.558029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:13.584688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1218/2418] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:20:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:20:13.590129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82048ef4-92b1-4abe-bc61-5b165955f78a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:13.601958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82048ef4-92b1-4abe-bc61-5b165955f78a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:13.615165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:13.615830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:13.616226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:13.616600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:13.618867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:13.622972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:13.685772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1201/2419] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:20:13 2022] POST /identity/v3/projects => generated 369 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:20:13.690057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:13.706183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5884a572-7665-411d-a7d5-ad5ac2c47902 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:13.818971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:13.821366 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:13.871803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:13.873657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:13.874199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:13.875921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:13.877834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:13.881402 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:13.932159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1219/2420] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:20:13 2022] POST /identity/v3/users => generated 285 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:20:13.936023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:13.952517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8242f6-96d9-460c-8a5d-ea859dbaf902 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:13.968786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:13.969629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:13.969907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:13.972191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:13.972887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:13.974557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:13.998728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1220/2421] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:20:13 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:20:14.000003 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0682ac2-026c-416f-a90b-b363c7edc2b5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:14.006863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:14.017672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:14.018351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:14.018635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:14.018909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] PATH_INFO: `/v3/projects/3d7275e32e964dcd8296d442d1bf1ea1` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:14.029628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df8837c-e9df-470b-ba2a-bbd35e5b9a2d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:14.029934 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d7275e32e964dcd8296d442d1bf1ea1)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:14.032971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:14.045446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:14.046371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:14.046638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:14.046914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:14.049377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:14.053080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:14.078043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1221/2422] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:20:14 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:20:14.083859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:14.144821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6619b9-c20f-43c7-974f-5b43171aa7b3 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:14.157634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d7275e32e964dcd8296d442d1bf1ea1', 'request_id': 'req-282ccde9-0e95-4383-8689-1b449f6c2c67'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:14.169612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d7275e32e964dcd8296d442d1bf1ea1', 'request_id': 'req-282ccde9-0e95-4383-8689-1b449f6c2c67'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:14.188291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d7275e32e964dcd8296d442d1bf1ea1', 'request_id': 'req-282ccde9-0e95-4383-8689-1b449f6c2c67'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:14.210592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:14.214286 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] The token cache is being invalidate because project 3d7275e32e964dcd8296d442d1bf1ea1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:20:14.214586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:14.217281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-282ccde9-0e95-4383-8689-1b449f6c2c67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:20:14.240561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1202/2423] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:20:13 2022] DELETE /identity/v3/projects/3d7275e32e964dcd8296d442d1bf1ea1 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:20:14.242894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:14.242894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:14.242894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:14.367229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] MFA Rules not processed for user `dea6fd8e6a374cd5b12ec7d99f285203`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:14.369339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:14.390477 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f53787-0f35-422c-9387-2e23bb50230c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:14.393420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f53787-0f35-422c-9387-2e23bb50230c admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:14.396785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f53787-0f35-422c-9387-2e23bb50230c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:14.397120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f53787-0f35-422c-9387-2e23bb50230c admin admin] PATH_INFO: `/v3/projects/f42af5112cc5497081e73002fc682d56/users/2027c6e926d141009c84eaa5ebb34a17/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:14.478120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f53787-0f35-422c-9387-2e23bb50230c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f42af5112cc5497081e73002fc682d56, user_id=2027c6e926d141009c84eaa5ebb34a17, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:14.482325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f53787-0f35-422c-9387-2e23bb50230c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:14.543132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1222/2424] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:20:14 2022] PUT /identity/v3/projects/f42af5112cc5497081e73002fc682d56/users/2027c6e926d141009c84eaa5ebb34a17/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:14.559999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:14.560431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:14.560877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:14.597261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:14.632148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1203/2425] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 391 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:14.633373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] MFA Rules not processed for user `2027c6e926d141009c84eaa5ebb34a17`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:14.634860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:14.637822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:14.643537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:14.688215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadcae3e-dee6-4348-b5c5-59caaa24edbb None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:14.833357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1223/2426] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:14 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:20:14.845462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:14.849842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:14.872597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:14.873274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:14.873999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:14.874279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:14.874933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:14.877859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:14.929310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1204/2427] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:20:14 2022] POST /identity/v3/projects => generated 379 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:20:14.947090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:15.058657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248e3cb7-40cf-4dfc-92ed-1e6267698f2e None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:15.132969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac13ff3-b95c-4543-b7a0-4f3f6c66d670 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:15.158403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:15.159182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:15.159678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:15.159799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:15.160388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:15.162450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:15.212804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:15.232543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1205/2428] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:20:14 2022] POST /identity/v3/users => generated 296 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:20:15.233995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:15.249038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:15.249704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:15.249995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:15.250277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:15.268820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff28874c-e942-4041-98d0-ec7c238cebf1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:15.283312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:15.283937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:15.284219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:15.284496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:15.285189 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:15.286796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:15.311885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1206/2429] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:20:15.315162 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:15.350462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9583f03f-8d7f-4af6-a066-1b819022685c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:15.386231 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:15.387889 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:15.388178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:15.388457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] PATH_INFO: `/v3/projects/997939da104a452fba57b826ccae88d7/users/392fdfae893e492fa9e08cc870b714a5/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:15.458033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:15.459513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:15.464181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] RBAC: Authorizing `identity:create_grant(project_id=997939da104a452fba57b826ccae88d7, user_id=392fdfae893e492fa9e08cc870b714a5, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:15.480019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:15.496382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1224/2430] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:14 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 650 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:15.555095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1207/2431] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:20:15 2022] PUT /identity/v3/projects/997939da104a452fba57b826ccae88d7/users/392fdfae893e492fa9e08cc870b714a5/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:20:15.559739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:15.579250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90623dcf-54e0-41f3-a783-176b9813bc48 service glance] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:15.740276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ddd838-0e7a-45a5-9f3c-3240cfdf4ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:15.740921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddd838-0e7a-45a5-9f3c-3240cfdf4ce3 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:15.741220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddd838-0e7a-45a5-9f3c-3240cfdf4ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:15.741636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ddd838-0e7a-45a5-9f3c-3240cfdf4ce3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:15.742643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ddd838-0e7a-45a5-9f3c-3240cfdf4ce3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:15.743971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ddd838-0e7a-45a5-9f3c-3240cfdf4ce3 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:15.769076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1225/2432] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:20:15.772057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:15.791020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce455f4-04c5-45f4-81ff-0133dbbba80d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:15.805558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:15.806432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:15.806922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:15.807353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] PATH_INFO: `/v3/projects/997939da104a452fba57b826ccae88d7/users/392fdfae893e492fa9e08cc870b714a5/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:15.811818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=997939da104a452fba57b826ccae88d7, user_id=392fdfae893e492fa9e08cc870b714a5, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:15.820870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:15.858598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1208/2433] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:20:15 2022] PUT /identity/v3/projects/997939da104a452fba57b826ccae88d7/users/392fdfae893e492fa9e08cc870b714a5/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:20:16.988135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7eef47-a46c-4e75-9af4-6ca1553aa08f None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:16.989339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7eef47-a46c-4e75-9af4-6ca1553aa08f None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:16.989955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7eef47-a46c-4e75-9af4-6ca1553aa08f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:17.030531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5d7eef47-a46c-4e75-9af4-6ca1553aa08f None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:17.032088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5d7eef47-a46c-4e75-9af4-6ca1553aa08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:17.163256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1226/2434] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:20:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:17.171457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:17.174699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:17.235653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:17.389588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d44f73-f118-4b96-93cb-587242a8bf49 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:17.409392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723cfa60-2d4c-4bbb-bbc3-4ed79d765b57 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:17.410185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723cfa60-2d4c-4bbb-bbc3-4ed79d765b57 service swift] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:17.410548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723cfa60-2d4c-4bbb-bbc3-4ed79d765b57 service swift] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:17.410950 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723cfa60-2d4c-4bbb-bbc3-4ed79d765b57 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:17.425271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723cfa60-2d4c-4bbb-bbc3-4ed79d765b57 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:17.427579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723cfa60-2d4c-4bbb-bbc3-4ed79d765b57 service swift] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:17.461440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1209/2435] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:20:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:20:22.345903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe74b2d-1cd4-4bfc-ba0c-421aab989870 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:22.346616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe74b2d-1cd4-4bfc-ba0c-421aab989870 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:22.346616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe74b2d-1cd4-4bfc-ba0c-421aab989870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:22.419641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-fbe74b2d-1cd4-4bfc-ba0c-421aab989870 None None] MFA Rules not processed for user `7014a1defd81461787318bc27d56b9aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:22.422539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fbe74b2d-1cd4-4bfc-ba0c-421aab989870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:22.553930 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:22.646649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1227/2436] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:22 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:27.440349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=22 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:28.536264 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:28.536907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:28.536907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:28.644676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] MFA Rules not processed for user `392fdfae893e492fa9e08cc870b714a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:28.646653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:28.855655 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1210/2437] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:28 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:28.992467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd4129-baaa-46d1-bd59-4dd30e2b1a19 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:28.993138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd4129-baaa-46d1-bd59-4dd30e2b1a19 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:28.993138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdd4129-baaa-46d1-bd59-4dd30e2b1a19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:28.998416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:29.001090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:29.040247 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-efdd4129-baaa-46d1-bd59-4dd30e2b1a19 None None] MFA Rules not processed for user `3d75f6b795064e00b20207e6cfc22cf1`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:29.045638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-efdd4129-baaa-46d1-bd59-4dd30e2b1a19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:29.201059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1a65d5-3475-43a3-94d4-2d62287ae042 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:29.220021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1228/2438] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:20:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:29.221800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be4ff25-f357-476b-977c-d9fd039c5841 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:29.222462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be4ff25-f357-476b-977c-d9fd039c5841 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:29.222732 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be4ff25-f357-476b-977c-d9fd039c5841 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:29.223042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be4ff25-f357-476b-977c-d9fd039c5841 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:29.317624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be4ff25-f357-476b-977c-d9fd039c5841 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:29.319437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be4ff25-f357-476b-977c-d9fd039c5841 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:29.332961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1211/2439] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:28 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:39.199946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=32 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:39.339931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=90 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:20:42.187147 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c8180-8729-48cf-88dc-b909e727c380 None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.187674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c8180-8729-48cf-88dc-b909e727c380 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.187674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5c8180-8729-48cf-88dc-b909e727c380 None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.189019 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1229/2440] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:20:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:20:42.194746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeaeda7-d3f5-45b7-ba33-de52f832daf8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.195173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeaeda7-d3f5-45b7-ba33-de52f832daf8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.195379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeaeda7-d3f5-45b7-ba33-de52f832daf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.217022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-1aeaeda7-d3f5-45b7-ba33-de52f832daf8 None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:42.218261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1aeaeda7-d3f5-45b7-ba33-de52f832daf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:42.227794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1212/2441] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:20:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:42.241573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5c8180-8729-48cf-88dc-b909e727c380 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:42.244823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7e5c8180-8729-48cf-88dc-b909e727c380 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:42.317508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5c8180-8729-48cf-88dc-b909e727c380 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:42.323299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c2dcf7-8e11-4cd1-a045-02919c39ee31 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:42.323991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c2dcf7-8e11-4cd1-a045-02919c39ee31 service glance] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.324343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c2dcf7-8e11-4cd1-a045-02919c39ee31 service glance] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.324685 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c2dcf7-8e11-4cd1-a045-02919c39ee31 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.335070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c2dcf7-8e11-4cd1-a045-02919c39ee31 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:42.336583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c2dcf7-8e11-4cd1-a045-02919c39ee31 service glance] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:42.347663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1230/2442] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:42.421381 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45b0d2d-f514-490e-a0c1-6c49c72fa0fb None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.421672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45b0d2d-f514-490e-a0c1-6c49c72fa0fb None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.421951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45b0d2d-f514-490e-a0c1-6c49c72fa0fb None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.423304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1213/2443] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:20:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:20:42.428383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3657ab59-1945-460b-8f11-06e0bb053cc1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.428674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3657ab59-1945-460b-8f11-06e0bb053cc1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.428978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3657ab59-1945-460b-8f11-06e0bb053cc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.449404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3657ab59-1945-460b-8f11-06e0bb053cc1 None None] MFA Rules not processed for user `16aab142c0a7445483caf19178d56062`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:42.450631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3657ab59-1945-460b-8f11-06e0bb053cc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:42.501643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1231/2444] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:20:42 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:20:42.513626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45b0d2d-f514-490e-a0c1-6c49c72fa0fb None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:42.515326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c45b0d2d-f514-490e-a0c1-6c49c72fa0fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:42.531961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45b0d2d-f514-490e-a0c1-6c49c72fa0fb None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:42.535525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0e919d-0810-495d-bde9-01ae672a0d65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:42.536117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0e919d-0810-495d-bde9-01ae672a0d65 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.536395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0e919d-0810-495d-bde9-01ae672a0d65 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.536691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0e919d-0810-495d-bde9-01ae672a0d65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.544902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0e919d-0810-495d-bde9-01ae672a0d65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:42.546490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0e919d-0810-495d-bde9-01ae672a0d65 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:42.555395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1214/2445] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:42.642305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd857d7-3192-483a-9ef8-c70329d40e2d None None] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.642590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd857d7-3192-483a-9ef8-c70329d40e2d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.642865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd857d7-3192-483a-9ef8-c70329d40e2d None None] PATH_INFO: `/` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.644239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1232/2446] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:20:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:20:42.649187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32fb4e-e542-4fdc-8ac0-1afbbfa8a1ad None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.650326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32fb4e-e542-4fdc-8ac0-1afbbfa8a1ad None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.650606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32fb4e-e542-4fdc-8ac0-1afbbfa8a1ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.669984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-eb32fb4e-e542-4fdc-8ac0-1afbbfa8a1ad None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:42.671163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-eb32fb4e-e542-4fdc-8ac0-1afbbfa8a1ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:42.680269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1215/2447] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:20:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:20:42.691050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd857d7-3192-483a-9ef8-c70329d40e2d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:42.702975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd857d7-3192-483a-9ef8-c70329d40e2d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:42.706555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:42.707143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:42.707430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:42.707701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:42.717739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:42.719262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:42.729981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1233/2448] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:20:43.832742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10464a5d-e9a6-453f-93a6-f9069c74d539 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:43.833323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10464a5d-e9a6-453f-93a6-f9069c74d539 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:43.833323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10464a5d-e9a6-453f-93a6-f9069c74d539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:43.866180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-10464a5d-e9a6-453f-93a6-f9069c74d539 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:43.869069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-10464a5d-e9a6-453f-93a6-f9069c74d539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:43.883698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1216/2449] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:20:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:20:43.899826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:43.914701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79e4a0e-2599-44f5-b717-228095a3f232 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:43.921514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:43.922372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:43.922751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:43.923388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:43.935066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:43.936943 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:43.966131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1234/2450] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:20:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:20:51.915999 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:51.916805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:51.916805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:51.940249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:20:51.941824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:51.949521 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1217/2451] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:20:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:20:51.962446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:51.974224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f766cfd8-1e2c-485f-a546-d86c3fa3dd2e service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:51.979132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa5bc83-7325-4739-9d77-708e58e7d345 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:51.979737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa5bc83-7325-4739-9d77-708e58e7d345 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:51.980000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa5bc83-7325-4739-9d77-708e58e7d345 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:51.980269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa5bc83-7325-4739-9d77-708e58e7d345 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:51.988866 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa5bc83-7325-4739-9d77-708e58e7d345 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:51.990598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa5bc83-7325-4739-9d77-708e58e7d345 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:51.999335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1235/2452] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:20:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:20:57.458320 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:20:57.460208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:57.582069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10be0759-3272-4e90-ad4b-16ea737f99d7 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:20:57.588282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:20:57.589059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:20:57.589598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:20:57.590129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:20:57.593993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:20:57.608833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:20:57.610860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:20:57.625345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1218/2453] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:20:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:21:01.995231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=67 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:03.062201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b3217-8ce2-44cb-904e-6e1e1eb59e7a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:03.062841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b3217-8ce2-44cb-904e-6e1e1eb59e7a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:03.062841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b3217-8ce2-44cb-904e-6e1e1eb59e7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:03.089086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-d40b3217-8ce2-44cb-904e-6e1e1eb59e7a None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:03.090354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d40b3217-8ce2-44cb-904e-6e1e1eb59e7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:03.100292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1236/2454] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:21:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:21:03.151301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:03.170537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089d1374-7422-4bd9-858d-7ac36243affa service placement] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:03.176959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:03.177821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:03.178328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:03.178801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:03.194946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:03.207092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:03.234369 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1219/2455] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:21:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:21:03.658624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca511d3-b1a1-4782-90b9-5cf644b79328 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:03.659017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca511d3-b1a1-4782-90b9-5cf644b79328 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:03.659394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca511d3-b1a1-4782-90b9-5cf644b79328 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:03.694040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-8ca511d3-b1a1-4782-90b9-5cf644b79328 None None] MFA Rules not processed for user `dea6fd8e6a374cd5b12ec7d99f285203`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:03.697257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8ca511d3-b1a1-4782-90b9-5cf644b79328 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:03.713566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1237/2456] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:21:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:21:13.223751 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=56 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:13.675411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:14.478064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:14.479173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:14.526197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42e9d1c-ad98-43f4-881c-4f2445654cd9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:14.526608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42e9d1c-ad98-43f4-881c-4f2445654cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:14.527006 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42e9d1c-ad98-43f4-881c-4f2445654cd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:14.545972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeebae3c-c21d-4093-96de-eab9ad5c3c70 service swift] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:14.550977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:14.551643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:14.551927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:14.552204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] PATH_INFO: `/v3/users/2027c6e926d141009c84eaa5ebb34a17` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:14.553403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2027c6e926d141009c84eaa5ebb34a17)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:14.553531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a42e9d1c-ad98-43f4-881c-4f2445654cd9 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:14.555429 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:14.555550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a42e9d1c-ad98-43f4-881c-4f2445654cd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:14.564605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1238/2457] 10.222.0.26 () {58 vars in 967 bytes} [Sat Oct 1 04:21:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:21:14.577783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42e9d1c-ad98-43f4-881c-4f2445654cd9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:14.593046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42e9d1c-ad98-43f4-881c-4f2445654cd9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:14.597985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25c7979-c995-4e98-8a64-0b311394e267 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:14.598579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25c7979-c995-4e98-8a64-0b311394e267 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:14.598856 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25c7979-c995-4e98-8a64-0b311394e267 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:14.599129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25c7979-c995-4e98-8a64-0b311394e267 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:14.604651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2027c6e926d141009c84eaa5ebb34a17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:14.605327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2027c6e926d141009c84eaa5ebb34a17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:14.609199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25c7979-c995-4e98-8a64-0b311394e267 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:14.610778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25c7979-c995-4e98-8a64-0b311394e267 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:14.613202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2027c6e926d141009c84eaa5ebb34a17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:14.613872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2027c6e926d141009c84eaa5ebb34a17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:14.620915 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1239/2458] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:21:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 01 04:21:14.622410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2027c6e926d141009c84eaa5ebb34a17'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:14.646266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1220/2459] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:21:14 2022] DELETE /identity/v3/users/2027c6e926d141009c84eaa5ebb34a17 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:21:14.651148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:14.651458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:14.652327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:14.672210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:14.673454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:14.739086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1240/2460] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:21:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:21:14.749055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:14.750581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:14.812370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1091878-31e3-4a45-92ff-67a858ffb166 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:14.816558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:14.817200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:14.817574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:14.817740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:14.828289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:14.829888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:14.840321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1221/2461] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:21:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:21:15.136635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:15.138809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:15.150950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37b138d-cc4a-421c-97de-fce2156ace7c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:15.156664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:15.157338 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:15.157799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:15.158082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] PATH_INFO: `/v3/projects/f42af5112cc5497081e73002fc682d56` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:15.159339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f42af5112cc5497081e73002fc682d56)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:15.161297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:15.209717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f42af5112cc5497081e73002fc682d56', 'request_id': 'req-8d39335c-e9cf-40a8-ba53-91ff674d03c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:15.219352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f42af5112cc5497081e73002fc682d56', 'request_id': 'req-8d39335c-e9cf-40a8-ba53-91ff674d03c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:15.225895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f42af5112cc5497081e73002fc682d56', 'request_id': 'req-8d39335c-e9cf-40a8-ba53-91ff674d03c5'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:15.236937 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] The token cache is being invalidate because project f42af5112cc5497081e73002fc682d56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:21:15.237346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:15.237945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:15.239600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1241/2462] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:21:15 2022] DELETE /identity/v3/projects/f42af5112cc5497081e73002fc682d56 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:21:16.941042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:16.942707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:16.958997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c17162-2b62-4924-8bb0-50cfb2d3fe18 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:16.983291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:17.050650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:17.051705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:17.052106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:17.053281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] PATH_INFO: `/v3/users/27dc4e15e61e44c8905e713a190a45d3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:17.054891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] RBAC: Authorizing `identity:delete_user(user_id=27dc4e15e61e44c8905e713a190a45d3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:17.057061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:17.116772 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27dc4e15e61e44c8905e713a190a45d3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.117481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27dc4e15e61e44c8905e713a190a45d3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.125051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27dc4e15e61e44c8905e713a190a45d3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.125695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27dc4e15e61e44c8905e713a190a45d3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.133147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6f356e99-d1e5-44ef-a138-5deff6d4beff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27dc4e15e61e44c8905e713a190a45d3'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.157677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1222/2463] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:21:16 2022] DELETE /identity/v3/users/27dc4e15e61e44c8905e713a190a45d3 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:21:17.400907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:17.405479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:17.426225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39335c-e9cf-40a8-ba53-91ff674d03c5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:17.490536 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:17.491204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:17.491482 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:17.491755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] PATH_INFO: `/v3/projects/6452abdab41846a5915b7ea5d81b9811` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:17.492863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6452abdab41846a5915b7ea5d81b9811)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:17.494792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:17.501789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:17.555454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6452abdab41846a5915b7ea5d81b9811', 'request_id': 'req-9929f247-b371-44e0-8cad-d81a979bc2e9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.563379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6452abdab41846a5915b7ea5d81b9811', 'request_id': 'req-9929f247-b371-44e0-8cad-d81a979bc2e9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.570460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6452abdab41846a5915b7ea5d81b9811', 'request_id': 'req-9929f247-b371-44e0-8cad-d81a979bc2e9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.582865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] The token cache is being invalidate because project 6452abdab41846a5915b7ea5d81b9811 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:21:17.583178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.583818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9929f247-b371-44e0-8cad-d81a979bc2e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:17.585331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1242/2464] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:21:17 2022] DELETE /identity/v3/projects/6452abdab41846a5915b7ea5d81b9811 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:21:17.592496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6c1a46-6df1-4ce3-a067-98c90842b478 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:17.592789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6c1a46-6df1-4ce3-a067-98c90842b478 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:17.593149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6c1a46-6df1-4ce3-a067-98c90842b478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:17.614742 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-8d6c1a46-6df1-4ce3-a067-98c90842b478 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:17.616170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8d6c1a46-6df1-4ce3-a067-98c90842b478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:17.698654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1223/2465] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:21:17.706981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5440f47c-975d-4ebe-addc-a9bf9e20a33d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:17.707533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5440f47c-975d-4ebe-addc-a9bf9e20a33d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:17.707819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5440f47c-975d-4ebe-addc-a9bf9e20a33d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:17.736398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-5440f47c-975d-4ebe-addc-a9bf9e20a33d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:17.737737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5440f47c-975d-4ebe-addc-a9bf9e20a33d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:17.758228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1243/2466] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:21:17.762713 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6c1a46-6df1-4ce3-a067-98c90842b478 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:17.775462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6c1a46-6df1-4ce3-a067-98c90842b478 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:17.781743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:17.782371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:17.782670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:17.782944 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:17.783589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:21:17.783896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:17.785758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:17.800641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1224/2467] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:21:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:21:17.806190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5440f47c-975d-4ebe-addc-a9bf9e20a33d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:17.822123 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5440f47c-975d-4ebe-addc-a9bf9e20a33d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:17.829316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:17.829929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:17.830209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:17.830636 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:17.831570 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:17.833596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:17.881671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1244/2468] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:21:17 2022] POST /identity/v3/projects => generated 389 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:21:17.885633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:17.897025 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a9d19d-d603-4c4d-956b-53bd4b2a38f4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:17.939238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:17.992433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:17.993953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:17.994443 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:17.994949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:17.995983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:17.998562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:18.033227 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1225/2469] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:21:17 2022] POST /identity/v3/users => generated 295 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:21:18.033392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:18.047162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d32cc01-af18-4a49-8352-09ec3967576a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:18.055339 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:18.056646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:18.057189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:18.057722 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:18.058793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:18.061094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:18.073455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1245/2470] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:21:18 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:21:18.078259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:18.092895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702bb0e-76a1-40b4-b7d8-98e0127a09d5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:18.098361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ef58d5-1c32-4c92-99f5-f4ba601bb135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:18.099061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ef58d5-1c32-4c92-99f5-f4ba601bb135 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:18.099337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ef58d5-1c32-4c92-99f5-f4ba601bb135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:18.099608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ef58d5-1c32-4c92-99f5-f4ba601bb135 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:18.100307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ef58d5-1c32-4c92-99f5-f4ba601bb135 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:18.101978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ef58d5-1c32-4c92-99f5-f4ba601bb135 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:18.119692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:18.121703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1226/2471] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:21:18 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:21:18.136080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82a0fbe-f877-4d89-929e-2fe2390207c0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:18.147993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:18.149692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:18.149982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:18.150264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] PATH_INFO: `/v3/projects/c8bd1c1423544107b55ba8dbbb66fae7/users/334928028aed4224af2241d7f7cb4a2a/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:18.171850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8bd1c1423544107b55ba8dbbb66fae7, user_id=334928028aed4224af2241d7f7cb4a2a, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:18.175303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:18.201055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1246/2472] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:21:18 2022] PUT /identity/v3/projects/c8bd1c1423544107b55ba8dbbb66fae7/users/334928028aed4224af2241d7f7cb4a2a/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:21:18.205400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:18.205814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:18.206163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:18.238570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] MFA Rules not processed for user `334928028aed4224af2241d7f7cb4a2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:18.239771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:18.341114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1227/2473] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:21:18 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:21:18.351976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:18.353948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:18.415447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd014f-7505-41eb-a97f-8dd1e5246a43 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:18.419872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:18.420595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:18.420947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:18.421300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:18.429996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:18.431467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:18.441158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1247/2474] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:21:18 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:21:28.336503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=53,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:28.436822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=75,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:35.663493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:35.665359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:35.731523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1a380a-2d3a-4905-abf5-aea5af48a29d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:35.735532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1a2010-a755-47fc-8166-98fba1bd818a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:35.736221 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1a2010-a755-47fc-8166-98fba1bd818a service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:35.736716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1a2010-a755-47fc-8166-98fba1bd818a service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:35.736908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1a2010-a755-47fc-8166-98fba1bd818a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:35.738737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fb1a2010-a755-47fc-8166-98fba1bd818a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:35.810466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1a2010-a755-47fc-8166-98fba1bd818a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:35.812067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1a2010-a755-47fc-8166-98fba1bd818a service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:35.821826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1228/2475] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:21:35 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:21:39.912871 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:40.029139 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def8ef3a-5ae7-4d1c-ad86-fd028f561a79 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:40.063427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:40.063427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:40.063427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:40.063427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] PATH_INFO: `/v3/users/334928028aed4224af2241d7f7cb4a2a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:40.065826 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] RBAC: Authorizing `identity:delete_user(user_id=334928028aed4224af2241d7f7cb4a2a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:40.068215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:40.163490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '334928028aed4224af2241d7f7cb4a2a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.173749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '334928028aed4224af2241d7f7cb4a2a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.175226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '334928028aed4224af2241d7f7cb4a2a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.175860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '334928028aed4224af2241d7f7cb4a2a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.188525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '334928028aed4224af2241d7f7cb4a2a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.222533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1248/2476] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:21:39 2022] DELETE /identity/v3/users/334928028aed4224af2241d7f7cb4a2a => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:21:40.226920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:40.227421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:40.227516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:40.262885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:40.264651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:40.343706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1229/2477] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:21:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:21:40.358021 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:40.361398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:40.438764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:40.440677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:40.441839 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af6dc31-14f1-442a-a2f6-9d4540dca27f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:40.445775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:40.446760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:40.446760 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:40.447109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:40.454217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c8c19-918b-4eda-8caa-86413f2e9ff5 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:40.456408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:40.457928 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0ba996-bf41-46f6-9325-82bac01ff669 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:40.458186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:40.458560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0ba996-bf41-46f6-9325-82bac01ff669 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:40.458974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0ba996-bf41-46f6-9325-82bac01ff669 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:40.459250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0ba996-bf41-46f6-9325-82bac01ff669 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:40.461151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ab0ba996-bf41-46f6-9325-82bac01ff669 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:40.469458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1249/2478] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:21:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:21:40.520434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:40.541982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0ba996-bf41-46f6-9325-82bac01ff669 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:40.543991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0ba996-bf41-46f6-9325-82bac01ff669 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:40.556215 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1230/2479] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:21:40 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:21:40.878308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:40.880231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:40.900947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f2cd0c-b5e9-4374-8719-010eb72fb801 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:40.906859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:40.907501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:40.907906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:40.908190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] PATH_INFO: `/v3/projects/c8bd1c1423544107b55ba8dbbb66fae7` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:40.909365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8bd1c1423544107b55ba8dbbb66fae7)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:40.911328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:40.929323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:40.963421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8bd1c1423544107b55ba8dbbb66fae7', 'request_id': 'req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.974608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8bd1c1423544107b55ba8dbbb66fae7', 'request_id': 'req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.981088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8bd1c1423544107b55ba8dbbb66fae7', 'request_id': 'req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.993260 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] The token cache is being invalidate because project c8bd1c1423544107b55ba8dbbb66fae7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:21:40.993557 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.994213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:21:40.995696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1250/2480] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:21:40 2022] DELETE /identity/v3/projects/c8bd1c1423544107b55ba8dbbb66fae7 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:21:42.399890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d470731-0175-478a-abc3-f1afdcd8ff43 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:42.400380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d470731-0175-478a-abc3-f1afdcd8ff43 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:42.400656 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d470731-0175-478a-abc3-f1afdcd8ff43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:42.423253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9d470731-0175-478a-abc3-f1afdcd8ff43 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:21:42.424837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9d470731-0175-478a-abc3-f1afdcd8ff43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:42.480602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1231/2481] 10.222.0.26 () {58 vars in 967 bytes} [Sat Oct 1 04:21:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:21:42.493432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:21:42.494825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:21:42.611628 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0cf7ac-6e3b-4ff8-9a97-9f9c4edae6ba admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:21:42.616880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6224abe0-7d92-40e7-8fac-826bb6e41ed3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:21:42.617546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6224abe0-7d92-40e7-8fac-826bb6e41ed3 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:21:42.617825 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6224abe0-7d92-40e7-8fac-826bb6e41ed3 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:21:42.618097 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6224abe0-7d92-40e7-8fac-826bb6e41ed3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:21:42.627741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6224abe0-7d92-40e7-8fac-826bb6e41ed3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:21:42.629459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6224abe0-7d92-40e7-8fac-826bb6e41ed3 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:21:42.639833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1251/2482] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:21:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:21:52.473948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=32 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:21:52.636474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=49,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:22:31.082722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:22:31.083312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:22:31.083312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:22:31.103424 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] MFA Rules not processed for user `7014a1defd81461787318bc27d56b9aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:22:31.104628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:22:31.171692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1232/2483] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:22:31 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:22:33.850603 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4a50f5-b7ba-4e8f-9cb4-d735b438de55 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:22:33.851171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4a50f5-b7ba-4e8f-9cb4-d735b438de55 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:22:33.851171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4a50f5-b7ba-4e8f-9cb4-d735b438de55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:22:33.870832 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9c4a50f5-b7ba-4e8f-9cb4-d735b438de55 None None] MFA Rules not processed for user `ccd631925a0a4f5ca8be5d3915778487`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:22:33.872042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9c4a50f5-b7ba-4e8f-9cb4-d735b438de55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:22:33.929731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1252/2484] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:22:33 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:22:41.164379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=25 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:22:42.804735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:22:42.806196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:22:42.861418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1011517d-c037-4d44-a682-49b6ead592d3 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:22:42.865141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d67396-ddfc-4375-bc90-6fe8a9974f8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:22:42.865747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d67396-ddfc-4375-bc90-6fe8a9974f8c service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:22:42.866029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d67396-ddfc-4375-bc90-6fe8a9974f8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:22:42.866305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d67396-ddfc-4375-bc90-6fe8a9974f8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:22:42.868039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-46d67396-ddfc-4375-bc90-6fe8a9974f8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:22:42.930276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4a50f5-b7ba-4e8f-9cb4-d735b438de55 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:22:42.940496 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4a50f5-b7ba-4e8f-9cb4-d735b438de55 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:22:42.941194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d67396-ddfc-4375-bc90-6fe8a9974f8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:22:42.942639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d67396-ddfc-4375-bc90-6fe8a9974f8c service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:22:42.944036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:22:42.944637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:22:42.944918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:22:42.945221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:22:42.946934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:22:42.953706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1233/2485] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:22:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:22:42.960469 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:22:42.961934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:22:42.973239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1253/2486] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:22:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:22:49.755894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775f8c4-6063-4e80-8916-3fd4f2e73809 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:22:49.756434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775f8c4-6063-4e80-8916-3fd4f2e73809 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:22:49.756434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2775f8c4-6063-4e80-8916-3fd4f2e73809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:22:49.775459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2775f8c4-6063-4e80-8916-3fd4f2e73809 None None] MFA Rules not processed for user `3caf1cc366434d418c474d7a1879faa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:22:49.776637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2775f8c4-6063-4e80-8916-3fd4f2e73809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:22:49.830696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1234/2487] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:22:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:22:52.968167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=31 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:22:59.823805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=79 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:01.033735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:01.035741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:01.050089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac183299-a0f7-48ac-a9a0-6fd7c5069465 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:01.055653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:01.056254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:01.056535 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:01.056811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:01.058544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:01.069120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:01.070750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:01.079742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1254/2488] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:01 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:04.795109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:04.795555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:04.795679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:04.818486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:04.819864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:04.828110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1235/2489] 10.222.0.26 () {58 vars in 967 bytes} [Sat Oct 1 04:23:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:23:04.883150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:04.894752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c89286-8cfd-40ea-a94d-3ddfc1cdb626 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:04.900262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:04.900865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:04.901201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:04.901440 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:04.910148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:04.912004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:04.920913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1255/2490] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:23:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:23:11.952093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:11.954292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:12.027188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c540e4-4d03-4ff1-8dc2-2ce996cf3d67 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:12.031900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:12.032587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:12.033062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:12.033344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] PATH_INFO: `/v3/users/3d75f6b795064e00b20207e6cfc22cf1` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:12.034559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d75f6b795064e00b20207e6cfc22cf1)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:12.036533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:12.094388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d75f6b795064e00b20207e6cfc22cf1'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.095115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d75f6b795064e00b20207e6cfc22cf1'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.102741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d75f6b795064e00b20207e6cfc22cf1'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.103467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d75f6b795064e00b20207e6cfc22cf1'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.113208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d75f6b795064e00b20207e6cfc22cf1'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.133006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1236/2491] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:23:11 2022] DELETE /identity/v3/users/3d75f6b795064e00b20207e6cfc22cf1 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:12.322439 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:12.333220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee449bb-5768-4b5c-a7a9-8f94fab429a8 service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:12.397539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:12.398174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:12.398455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:12.398728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] PATH_INFO: `/v3/projects/1d29e2e750f948e1b0ccd4117c0d347a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:12.399833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d29e2e750f948e1b0ccd4117c0d347a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:12.401786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:12.447835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d29e2e750f948e1b0ccd4117c0d347a', 'request_id': 'req-470d47fa-7897-4b41-9bdf-a64a4184dccf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.454634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d29e2e750f948e1b0ccd4117c0d347a', 'request_id': 'req-470d47fa-7897-4b41-9bdf-a64a4184dccf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.460830 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d29e2e750f948e1b0ccd4117c0d347a', 'request_id': 'req-470d47fa-7897-4b41-9bdf-a64a4184dccf'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.471638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] The token cache is being invalidate because project 1d29e2e750f948e1b0ccd4117c0d347a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:23:12.471928 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.472568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.474076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1256/2492] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:23:12 2022] DELETE /identity/v3/projects/1d29e2e750f948e1b0ccd4117c0d347a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:12.478123 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:12.479654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:12.493601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca379d9-af2d-4e05-8adf-27471bcdc29e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:12.552437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:12.553020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:12.553305 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:12.553583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] PATH_INFO: `/v3/users/61e29adae78246189f15e257e9c2f98b` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:12.554723 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] RBAC: Authorizing `identity:delete_user(user_id=61e29adae78246189f15e257e9c2f98b)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:12.556641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:12.571970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:12.605406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61e29adae78246189f15e257e9c2f98b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.606065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61e29adae78246189f15e257e9c2f98b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.612942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61e29adae78246189f15e257e9c2f98b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.613570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61e29adae78246189f15e257e9c2f98b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.620471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61e29adae78246189f15e257e9c2f98b'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.638823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1237/2493] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:23:12 2022] DELETE /identity/v3/users/61e29adae78246189f15e257e9c2f98b => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:12.816793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:12.818335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:12.835148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470d47fa-7897-4b41-9bdf-a64a4184dccf admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:12.899619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:12.900173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:12.900451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:12.900725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] PATH_INFO: `/v3/projects/0187d00cc2bb49b3812b571c884fd809` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:12.901895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0187d00cc2bb49b3812b571c884fd809)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:12.903818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:12.910058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:12.952013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0187d00cc2bb49b3812b571c884fd809', 'request_id': 'req-3c0f62af-6838-489f-bebd-206d6a07d1c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.958917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0187d00cc2bb49b3812b571c884fd809', 'request_id': 'req-3c0f62af-6838-489f-bebd-206d6a07d1c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.965441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0187d00cc2bb49b3812b571c884fd809', 'request_id': 'req-3c0f62af-6838-489f-bebd-206d6a07d1c4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.975765 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] The token cache is being invalidate because project 0187d00cc2bb49b3812b571c884fd809 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:23:12.976056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.976691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:12.978112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1257/2494] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:23:12 2022] DELETE /identity/v3/projects/0187d00cc2bb49b3812b571c884fd809 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:14.179441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:14.180860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:14.357202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c281de8-712e-443e-a943-25c89cb7232e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:14.403479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:14.403996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:14.404272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:14.404541 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:14.406239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:14.463340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:14.464750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:14.474566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1238/2495] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:14 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:20.455191 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:20.456670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:20.571518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0f62af-6838-489f-bebd-206d6a07d1c4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:20.577653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:20.578410 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:20.578707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:20.578985 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:20.580761 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:20.599637 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:20.602377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:20.618948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1258/2496] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:24.136415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:24.142847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:24.156211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a26ce6-8c8f-4e3a-bf27-1e031bfb1c6e service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:24.161597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:24.162213 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:24.163042 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:24.163409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:24.165113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:24.222490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:24.223988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:24.233527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1239/2497] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:24 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:30.609976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=56,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:34.229301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=27 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:34.397970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:34.399526 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:34.453913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d65302-ea8a-4193-b176-ebb6ef2003e9 service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:34.457509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:34.458134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:34.458414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:34.458686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:34.460893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:34.524389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:34.525853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:34.535488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1259/2498] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:37.827357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:37.828911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:37.842896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9bc154-3550-4d0c-925d-3bbc47a4b8f1 service nova] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:37.847468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:37.848092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:37.848367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:37.848651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] PATH_INFO: `/v3/users/7014a1defd81461787318bc27d56b9aa` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:37.849768 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] RBAC: Authorizing `identity:delete_user(user_id=7014a1defd81461787318bc27d56b9aa)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:37.852400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:37.901863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7014a1defd81461787318bc27d56b9aa'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:37.902572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7014a1defd81461787318bc27d56b9aa'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:37.909989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7014a1defd81461787318bc27d56b9aa'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:37.910680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7014a1defd81461787318bc27d56b9aa'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:37.917983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-23db99ae-4d7b-4f22-94aa-59583e2b842e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7014a1defd81461787318bc27d56b9aa'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:37.936582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1240/2499] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:23:37 2022] DELETE /identity/v3/users/7014a1defd81461787318bc27d56b9aa => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:38.124083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.136103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e9125c-eca0-49d8-94f0-75f2eeaa255a service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.196804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.197486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.197768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.198045 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] PATH_INFO: `/v3/projects/839c91c5262a49768c3e937b5a034137` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.199183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=839c91c5262a49768c3e937b5a034137)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.201208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.253303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '839c91c5262a49768c3e937b5a034137', 'request_id': 'req-143450a9-7938-46f1-aedc-5af2177791ac'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:38.256255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:38.262033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '839c91c5262a49768c3e937b5a034137', 'request_id': 'req-143450a9-7938-46f1-aedc-5af2177791ac'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:38.270703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '839c91c5262a49768c3e937b5a034137', 'request_id': 'req-143450a9-7938-46f1-aedc-5af2177791ac'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:38.282961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] The token cache is being invalidate because project 839c91c5262a49768c3e937b5a034137 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:23:38.283255 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:38.283921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-143450a9-7938-46f1-aedc-5af2177791ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:38.285434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1260/2500] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:23:38 2022] DELETE /identity/v3/projects/839c91c5262a49768c3e937b5a034137 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:23:38.297858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dda4b4-075b-459e-bf35-9a7883fd2ffa None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.298316 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dda4b4-075b-459e-bf35-9a7883fd2ffa None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.298580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dda4b4-075b-459e-bf35-9a7883fd2ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.317800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-03dda4b4-075b-459e-bf35-9a7883fd2ffa None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:38.318979 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-03dda4b4-075b-459e-bf35-9a7883fd2ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:38.375916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1241/2501] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:38.384642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613320a5-f7c5-480f-bae6-197761070a58 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.384951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613320a5-f7c5-480f-bae6-197761070a58 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.385240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613320a5-f7c5-480f-bae6-197761070a58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.405481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-613320a5-f7c5-480f-bae6-197761070a58 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:38.406681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-613320a5-f7c5-480f-bae6-197761070a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:38.414566 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1261/2502] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:38.418811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dda4b4-075b-459e-bf35-9a7883fd2ffa None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.428289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dda4b4-075b-459e-bf35-9a7883fd2ffa None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.433208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.433805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.434085 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.434361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.435011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:23:38.435312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.436885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.448771 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1242/2503] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:23:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:23:38.452888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613320a5-f7c5-480f-bae6-197761070a58 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.462505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613320a5-f7c5-480f-bae6-197761070a58 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.467133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.468195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.468483 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.468753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.469419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.471342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.497724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1262/2504] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:23:38 2022] POST /identity/v3/projects => generated 367 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:38.501865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.513372 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d59104-e3c2-4f89-abf3-e68898858228 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.567791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.568352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.568639 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.568891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.570212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.572088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.579928 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:38.595537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1243/2505] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:23:38 2022] POST /identity/v3/users => generated 284 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:38.599823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.609829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053b43b3-433d-4acf-9ed6-869cdad7b280 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.614347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.614932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.615206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.615471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.616136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.617782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.627312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1263/2506] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:38 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:38.631090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.641531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18e5855-c85f-4edd-aee2-a94bf686ce17 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.645981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55da94b8-297b-4079-803b-0cb6af9db385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.646847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55da94b8-297b-4079-803b-0cb6af9db385 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.647124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55da94b8-297b-4079-803b-0cb6af9db385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.648029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55da94b8-297b-4079-803b-0cb6af9db385 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.648737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55da94b8-297b-4079-803b-0cb6af9db385 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.650324 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55da94b8-297b-4079-803b-0cb6af9db385 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.659871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1244/2507] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:38.664126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.673739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078613d2-0136-422b-ba34-e585a045685c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.678156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.678945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.679389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.679667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] PATH_INFO: `/v3/projects/5b22ab06cb574340afcca170a2e4cbe0/users/1f57a3f97f2a47b9bdeb78a492efba32/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.695945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b22ab06cb574340afcca170a2e4cbe0, user_id=1f57a3f97f2a47b9bdeb78a492efba32, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.698661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.711725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1264/2508] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:23:38 2022] PUT /identity/v3/projects/5b22ab06cb574340afcca170a2e4cbe0/users/1f57a3f97f2a47b9bdeb78a492efba32/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:23:38.722900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc7f186-a9cb-46a5-8186-719e7a6000e3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.723184 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc7f186-a9cb-46a5-8186-719e7a6000e3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.723457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc7f186-a9cb-46a5-8186-719e7a6000e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.751972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-7dc7f186-a9cb-46a5-8186-719e7a6000e3 None None] MFA Rules not processed for user `1f57a3f97f2a47b9bdeb78a492efba32`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:38.753190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-7dc7f186-a9cb-46a5-8186-719e7a6000e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:38.816746 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1245/2509] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:38 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:38.827961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:38.829641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:38.884372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0323f5-567e-40a3-87a3-b56333a46a81 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:38.887955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:38.888551 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:38.888831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:38.889134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:38.898071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:38.899485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:38.908306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1265/2510] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:38 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:44.915582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:44.916067 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:44.916158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:44.935452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:44.936637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:44.993515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1246/2511] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:23:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:45.006195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:45.007675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:45.083749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cd950b-ee7c-4a96-b56f-f35eb2b0cdbe service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:45.089455 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:45.090075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:45.090357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:45.090630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:45.098988 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:45.100575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:45.108727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1266/2512] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:23:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:23:45.661667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:45.663417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:45.769593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cadbc2-260d-40d4-9d54-2b8aeb5f0030 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:45.777635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb0a4de-1b7c-4d8f-8591-c8743ccc53d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:45.778258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb0a4de-1b7c-4d8f-8591-c8743ccc53d4 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:45.778534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb0a4de-1b7c-4d8f-8591-c8743ccc53d4 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:45.778801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb0a4de-1b7c-4d8f-8591-c8743ccc53d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:45.780560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5fb0a4de-1b7c-4d8f-8591-c8743ccc53d4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:45.796872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb0a4de-1b7c-4d8f-8591-c8743ccc53d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:45.798519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb0a4de-1b7c-4d8f-8591-c8743ccc53d4 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:45.803605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:45.815024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1247/2513] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:47.564657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:47.576626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:47.649586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fb714-daab-4696-be69-572b1855e7db service swift] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:47.655601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:47.656358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:47.656723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:47.657087 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] PATH_INFO: `/v3/users/1f57a3f97f2a47b9bdeb78a492efba32` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:47.658467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f57a3f97f2a47b9bdeb78a492efba32)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:47.660624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:47.726132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f57a3f97f2a47b9bdeb78a492efba32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:47.744513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f57a3f97f2a47b9bdeb78a492efba32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:47.745700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f57a3f97f2a47b9bdeb78a492efba32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:47.747916 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f57a3f97f2a47b9bdeb78a492efba32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:47.772026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f57a3f97f2a47b9bdeb78a492efba32'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:47.794641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1267/2514] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:23:47 2022] DELETE /identity/v3/users/1f57a3f97f2a47b9bdeb78a492efba32 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:47.799394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:47.800323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:47.800604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:47.822020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:47.823238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:47.891249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1248/2515] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:47.902085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:47.903647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:47.987197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f53b21-e81c-4119-921b-4a144cf761c6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:47.993128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a22a56-1631-43c0-972d-10380fbfcff4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:47.993758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a22a56-1631-43c0-972d-10380fbfcff4 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:47.994035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a22a56-1631-43c0-972d-10380fbfcff4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:47.994307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a22a56-1631-43c0-972d-10380fbfcff4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.005282 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a22a56-1631-43c0-972d-10380fbfcff4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:48.006712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a22a56-1631-43c0-972d-10380fbfcff4 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:48.023847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1268/2516] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:48.354069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:48.355722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:48.375805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce459bb9-18db-4c13-86f5-2ec8295ba52c None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:48.381691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:48.382353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:48.383157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:48.383595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] PATH_INFO: `/v3/projects/5b22ab06cb574340afcca170a2e4cbe0` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.384841 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b22ab06cb574340afcca170a2e4cbe0)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:48.386961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:48.444716 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b22ab06cb574340afcca170a2e4cbe0', 'request_id': 'req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:48.452005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b22ab06cb574340afcca170a2e4cbe0', 'request_id': 'req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:48.461116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b22ab06cb574340afcca170a2e4cbe0', 'request_id': 'req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:48.479678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] The token cache is being invalidate because project 5b22ab06cb574340afcca170a2e4cbe0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:23:48.480258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:48.481409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-9e7081b3-6fd3-4c80-b394-7b4a67fcdae7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:48.482996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1249/2517] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:23:48 2022] DELETE /identity/v3/projects/5b22ab06cb574340afcca170a2e4cbe0 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:48.490043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c5428e-81a7-422c-8a7a-b830d54daaeb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:48.490676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c5428e-81a7-422c-8a7a-b830d54daaeb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:48.491217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c5428e-81a7-422c-8a7a-b830d54daaeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.519869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-04c5428e-81a7-422c-8a7a-b830d54daaeb None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:48.519869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-04c5428e-81a7-422c-8a7a-b830d54daaeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:48.561608 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:48.642350 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1269/2518] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:48.650182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b728-a284-4ee6-b84d-acfd2f0b651b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:48.650809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b728-a284-4ee6-b84d-acfd2f0b651b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:48.651894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b728-a284-4ee6-b84d-acfd2f0b651b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.678069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5d32b728-a284-4ee6-b84d-acfd2f0b651b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:48.679882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5d32b728-a284-4ee6-b84d-acfd2f0b651b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:48.695124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c5428e-81a7-422c-8a7a-b830d54daaeb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:48.696220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1250/2519] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:48.707186 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c5428e-81a7-422c-8a7a-b830d54daaeb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:48.713775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:48.714403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:48.714675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:48.714950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.715605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:23:48.715946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:48.719201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:48.732225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1270/2520] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:23:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:23:48.736336 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32b728-a284-4ee6-b84d-acfd2f0b651b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:48.747765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32b728-a284-4ee6-b84d-acfd2f0b651b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:48.753804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:48.754420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:48.754697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:48.754971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.755649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:48.758738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:48.797991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1251/2521] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:23:48 2022] POST /identity/v3/projects => generated 389 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:48.800659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:48.824868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe0a73a-7711-446c-a2ef-f8d04387998a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:48.904809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:48.906445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:48.906875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:48.907257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.908804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:48.911274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:48.947788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1271/2522] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:23:48 2022] POST /identity/v3/users => generated 295 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:48.948004 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:48.962536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43dd24a-e14e-447e-aadb-8a4aeafe03db admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:48.984786 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:48.985448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:48.985730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:48.986065 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:48.986722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:48.990275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.010733 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.011857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1252/2523] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:48 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:49.026573 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb93cb9-a079-42eb-8863-578655544af9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.036731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5cdb06-16f3-474f-93dd-32ad41981dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.037709 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5cdb06-16f3-474f-93dd-32ad41981dac admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.038129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5cdb06-16f3-474f-93dd-32ad41981dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.038849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5cdb06-16f3-474f-93dd-32ad41981dac admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.039789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5cdb06-16f3-474f-93dd-32ad41981dac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.042563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5cdb06-16f3-474f-93dd-32ad41981dac admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.062964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1272/2524] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:49 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:49.063119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.076181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0233ad-a8de-45e3-8aa8-c36d12a23c88 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.082801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.083519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.083854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.084135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] PATH_INFO: `/v3/projects/3782c7948da042d594d7acd1f35c2c61/users/f842e6819ba644b08ef7cb6c2bc0ee86/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.112892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] RBAC: Authorizing `identity:create_grant(project_id=3782c7948da042d594d7acd1f35c2c61, user_id=f842e6819ba644b08ef7cb6c2bc0ee86, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.119505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.164513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1253/2525] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:23:49 2022] PUT /identity/v3/projects/3782c7948da042d594d7acd1f35c2c61/users/f842e6819ba644b08ef7cb6c2bc0ee86/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:49.174100 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8cf2f-1dba-4496-a3fa-099baa0748ae None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.174488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8cf2f-1dba-4496-a3fa-099baa0748ae None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.174872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e8cf2f-1dba-4496-a3fa-099baa0748ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.227840 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-92e8cf2f-1dba-4496-a3fa-099baa0748ae None None] MFA Rules not processed for user `f842e6819ba644b08ef7cb6c2bc0ee86`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:49.229476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-92e8cf2f-1dba-4496-a3fa-099baa0748ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:49.343854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1273/2526] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:49 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:49.345987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.371775 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:49.427707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6a7efe-48ff-4ba7-97c6-c5915a44bbde admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.433035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.433769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.434118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.434465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.435214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.437300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.471268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1254/2527] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:23:49 2022] POST /identity/v3/projects => generated 389 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:49.472357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e8cf2f-1dba-4496-a3fa-099baa0748ae None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.485156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e8cf2f-1dba-4496-a3fa-099baa0748ae None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.514520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:49.553513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.554321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.555353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.555724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.556447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.558486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.583744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1274/2528] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:23:49 2022] POST /identity/v3/users => generated 301 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:49.589261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.600285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daab6393-6dac-42c7-933a-bb5f3005e005 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.605572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.606185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.606465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.608701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.608701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.609082 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.624377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1255/2529] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:49 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:49.625531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.636403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab880e2c-61be-443a-b03f-8ea8af03b7d5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.642073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.642739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.643332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.643332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] PATH_INFO: `/v3/projects/71b6403685994e308344dc2be030975e/users/996bdbaf07bc4396b3c3842a7cba6668/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.660855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71b6403685994e308344dc2be030975e, user_id=996bdbaf07bc4396b3c3842a7cba6668, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.663804 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.681589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1275/2530] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:23:49 2022] PUT /identity/v3/projects/71b6403685994e308344dc2be030975e/users/996bdbaf07bc4396b3c3842a7cba6668/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:49.685511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.699418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e1d8c2-e132-4168-bb13-e4246035b757 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.765114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98697d4-2499-4a52-90b7-fa8ec2c22d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.765774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98697d4-2499-4a52-90b7-fa8ec2c22d99 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.766057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98697d4-2499-4a52-90b7-fa8ec2c22d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.766326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98697d4-2499-4a52-90b7-fa8ec2c22d99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.766960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98697d4-2499-4a52-90b7-fa8ec2c22d99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.768561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98697d4-2499-4a52-90b7-fa8ec2c22d99 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.783295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1256/2531] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:49 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:49.784098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.794920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23b9d09-76c3-412b-a422-5f495d0c55c2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:49.800057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:49.801103 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.801399 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.801705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] PATH_INFO: `/v3/projects/71b6403685994e308344dc2be030975e/users/996bdbaf07bc4396b3c3842a7cba6668/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.804093 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] RBAC: Authorizing `identity:create_grant(project_id=71b6403685994e308344dc2be030975e, user_id=996bdbaf07bc4396b3c3842a7cba6668, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:49.806948 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:49.829132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1276/2532] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:23:49 2022] PUT /identity/v3/projects/71b6403685994e308344dc2be030975e/users/996bdbaf07bc4396b3c3842a7cba6668/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:23:49.834830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4835b62b-8edd-4ab2-9cc5-28805e70006a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:49.835122 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4835b62b-8edd-4ab2-9cc5-28805e70006a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:49.835403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4835b62b-8edd-4ab2-9cc5-28805e70006a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:49.868303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-4835b62b-8edd-4ab2-9cc5-28805e70006a None None] MFA Rules not processed for user `996bdbaf07bc4396b3c3842a7cba6668`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:49.869614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4835b62b-8edd-4ab2-9cc5-28805e70006a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:49.960128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1257/2533] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:49 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:23:49.969885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:49.972090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:50.049726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5842d6dd-ad2a-44f1-b969-0bbfa46fd02d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:50.053585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f773ef8f-9f12-41f8-b8ef-0fb9e49d79cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:50.054386 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773ef8f-9f12-41f8-b8ef-0fb9e49d79cb service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:50.054925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773ef8f-9f12-41f8-b8ef-0fb9e49d79cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:50.055307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773ef8f-9f12-41f8-b8ef-0fb9e49d79cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:50.068002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f773ef8f-9f12-41f8-b8ef-0fb9e49d79cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:50.069680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f773ef8f-9f12-41f8-b8ef-0fb9e49d79cb service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:50.081198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1277/2534] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:49 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:50.252574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4835b62b-8edd-4ab2-9cc5-28805e70006a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:50.293794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:50.366277 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4835b62b-8edd-4ab2-9cc5-28805e70006a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:50.374857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d6748-963e-4479-a183-1023c78068af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:50.375661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d6748-963e-4479-a183-1023c78068af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:50.375977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d6748-963e-4479-a183-1023c78068af admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:50.376326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d6748-963e-4479-a183-1023c78068af admin admin] PATH_INFO: `/v3/users/f842e6819ba644b08ef7cb6c2bc0ee86` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:50.377719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d6748-963e-4479-a183-1023c78068af admin admin] RBAC: Authorizing `identity:delete_user(user_id=f842e6819ba644b08ef7cb6c2bc0ee86)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:50.380687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d6748-963e-4479-a183-1023c78068af admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:50.446781 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f842e6819ba644b08ef7cb6c2bc0ee86'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:50.447560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f842e6819ba644b08ef7cb6c2bc0ee86'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:50.457739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f842e6819ba644b08ef7cb6c2bc0ee86'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:50.461353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f842e6819ba644b08ef7cb6c2bc0ee86'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:50.471659 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f842e6819ba644b08ef7cb6c2bc0ee86'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:50.511633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1258/2535] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:23:50 2022] DELETE /identity/v3/users/f842e6819ba644b08ef7cb6c2bc0ee86 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:50.513230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:50.513713 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:50.513997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:50.547963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:50.549491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:50.651887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1278/2536] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:23:50.662076 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:50.664884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:50.749370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d6748-963e-4479-a183-1023c78068af admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:50.753478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:50.754164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:50.754444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:50.754718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:50.765799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:50.767313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:50.778958 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1259/2537] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:23:51.396708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:51.399870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:51.436875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261b3c6a-8bfb-40cd-9b44-d58d899de511 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:51.445980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:51.446741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:51.447082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:51.447437 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] PATH_INFO: `/v3/projects/3782c7948da042d594d7acd1f35c2c61` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:51.448670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3782c7948da042d594d7acd1f35c2c61)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:51.450714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:51.470750 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:51.514730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3782c7948da042d594d7acd1f35c2c61', 'request_id': 'req-082709a0-5aa9-4655-8a79-aa544b1c8059'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.533274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3782c7948da042d594d7acd1f35c2c61', 'request_id': 'req-082709a0-5aa9-4655-8a79-aa544b1c8059'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.540902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3782c7948da042d594d7acd1f35c2c61', 'request_id': 'req-082709a0-5aa9-4655-8a79-aa544b1c8059'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.557431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] The token cache is being invalidate because project 3782c7948da042d594d7acd1f35c2c61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:23:51.557736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.558470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.560297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1279/2538] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:23:51 2022] DELETE /identity/v3/projects/3782c7948da042d594d7acd1f35c2c61 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:51.564441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:51.577230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b2e48c-2f9e-4075-9e55-9e11094349bb service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:51.625982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:51.688827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:51.689636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:51.689914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:51.690186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] PATH_INFO: `/v3/users/996bdbaf07bc4396b3c3842a7cba6668` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:51.692512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] RBAC: Authorizing `identity:delete_user(user_id=996bdbaf07bc4396b3c3842a7cba6668)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:51.694465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:51.799051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '996bdbaf07bc4396b3c3842a7cba6668'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.799709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '996bdbaf07bc4396b3c3842a7cba6668'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.809118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '996bdbaf07bc4396b3c3842a7cba6668'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.809833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '996bdbaf07bc4396b3c3842a7cba6668'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.819607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a62293ed-23f5-4b39-9f73-ce28a859e514 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '996bdbaf07bc4396b3c3842a7cba6668'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:51.850628 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1260/2539] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:23:51 2022] DELETE /identity/v3/users/996bdbaf07bc4396b3c3842a7cba6668 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:23:52.335142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:52.341887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:52.371717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082709a0-5aa9-4655-8a79-aa544b1c8059 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:52.528567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:52.529337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:52.529865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:52.530863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] PATH_INFO: `/v3/projects/71b6403685994e308344dc2be030975e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:52.532609 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] RBAC: Authorizing `identity:delete_project(project_id=71b6403685994e308344dc2be030975e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:52.542572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:52.720375 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71b6403685994e308344dc2be030975e', 'request_id': 'req-b18e724c-9898-41e9-b74a-929f49f8546d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:52.735679 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71b6403685994e308344dc2be030975e', 'request_id': 'req-b18e724c-9898-41e9-b74a-929f49f8546d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:52.746426 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71b6403685994e308344dc2be030975e', 'request_id': 'req-b18e724c-9898-41e9-b74a-929f49f8546d'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:52.763404 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] The token cache is being invalidate because project 71b6403685994e308344dc2be030975e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:23:52.763802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:52.764569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b18e724c-9898-41e9-b74a-929f49f8546d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:52.769195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1280/2540] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:23:52 2022] DELETE /identity/v3/projects/71b6403685994e308344dc2be030975e => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:52.774445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74e831d-a3c1-4b10-afa9-311e51d7854f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:52.775433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74e831d-a3c1-4b10-afa9-311e51d7854f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:52.775914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74e831d-a3c1-4b10-afa9-311e51d7854f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:52.811253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b74e831d-a3c1-4b10-afa9-311e51d7854f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:52.812846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b74e831d-a3c1-4b10-afa9-311e51d7854f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:52.944807 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1261/2541] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:52.947166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06db43-78c1-4d48-af32-e6188e8205be None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:52.947458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06db43-78c1-4d48-af32-e6188e8205be None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:52.947739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06db43-78c1-4d48-af32-e6188e8205be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:52.993067 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0e06db43-78c1-4d48-af32-e6188e8205be None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:52.994459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e06db43-78c1-4d48-af32-e6188e8205be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:53.011324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1281/2542] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:53.015719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74e831d-a3c1-4b10-afa9-311e51d7854f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:53.032332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74e831d-a3c1-4b10-afa9-311e51d7854f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:53.044667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:53.046187 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.046536 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.048015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.049557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:23:53.050141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:53.053485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:53.072981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1262/2543] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:23:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:23:53.077604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06db43-78c1-4d48-af32-e6188e8205be None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:53.092376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06db43-78c1-4d48-af32-e6188e8205be None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:53.098357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:53.098520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.098811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.099150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.099852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:53.102369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:53.137293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1282/2544] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:23:53 2022] POST /identity/v3/projects => generated 375 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:23:53.141137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:53.155653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9090b160-05ee-4265-855c-b44896100e14 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:53.294971 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:53.345365 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:53.346032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.346422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.346980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.348408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:53.351681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:53.393826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1263/2545] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:23:53 2022] POST /identity/v3/users => generated 288 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:23:53.394911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:53.408389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad4d37d-070d-4e3e-a49b-64a37f89a983 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:53.415675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:53.416868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.417383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.417888 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.419281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:53.430962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:53.434417 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:53.455344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1283/2546] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:53 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:53.459479 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:53.471621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61ef129-3cbf-4bcf-ab9d-f7338ced4955 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:53.480461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9a00ef-4763-4653-a80a-f9ba92d24843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:53.481131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a00ef-4763-4653-a80a-f9ba92d24843 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.481415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a00ef-4763-4653-a80a-f9ba92d24843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.481686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9a00ef-4763-4653-a80a-f9ba92d24843 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.482337 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9a00ef-4763-4653-a80a-f9ba92d24843 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:53.484184 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9a00ef-4763-4653-a80a-f9ba92d24843 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:53.524591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1264/2547] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:53 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:53.526127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:53.540096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d76d316-ba50-47ab-9e44-1a406336caf8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:53.547308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:53.548057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.548408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.548755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] PATH_INFO: `/v3/projects/89afa93271f944eb9324505f73422d54/users/7c6abe27ceac49a1abf3dcc083c5006d/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.570876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89afa93271f944eb9324505f73422d54, user_id=7c6abe27ceac49a1abf3dcc083c5006d, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:53.574095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:53.593995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1284/2548] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:23:53 2022] PUT /identity/v3/projects/89afa93271f944eb9324505f73422d54/users/7c6abe27ceac49a1abf3dcc083c5006d/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:53.605270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b79773a-28c3-4add-b1aa-eb425bb06ab3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.605555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b79773a-28c3-4add-b1aa-eb425bb06ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.605824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b79773a-28c3-4add-b1aa-eb425bb06ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.643396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9b79773a-28c3-4add-b1aa-eb425bb06ab3 None None] MFA Rules not processed for user `7c6abe27ceac49a1abf3dcc083c5006d`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:53.644769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9b79773a-28c3-4add-b1aa-eb425bb06ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:53.750370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1265/2549] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:53 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:53.764857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:53.769800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:53.850312 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf144e9-8879-4ac6-884f-27ac07885465 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:53.855740 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:53.856420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:53.857042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:53.857181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:53.881651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:53.883308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:53.894219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1285/2550] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:53 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:54.004204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:54.004705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:54.004844 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:54.034246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] MFA Rules not processed for user `dea6fd8e6a374cd5b12ec7d99f285203`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:54.035491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:54.139239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1266/2551] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:54.569359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:54.571822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:54.639036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edcaa67-ea55-48be-a966-fec462c90d05 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:54.645246 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:54.646435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:54.646782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:54.647117 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:54.649078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:54.659516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:54.660953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:54.670107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1286/2552] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:55.247185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:55.248584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:55.331988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:55.354985 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0777c-41ec-4e0c-af93-f9855c8126c6 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:55.360410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:55.361148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:55.361497 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:55.361833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:55.365542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:55.378790 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:55.384988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:55.392115 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1267/2553] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:57.187009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:57.188480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:57.200063 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d822cb02-8e3b-414a-8fa5-641df1ac6cd2 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:57.204541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8830a80a-fd37-42d9-85ed-8456fae63af0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:57.205268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830a80a-fd37-42d9-85ed-8456fae63af0 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:57.205611 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830a80a-fd37-42d9-85ed-8456fae63af0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:57.205945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8830a80a-fd37-42d9-85ed-8456fae63af0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:57.207790 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8830a80a-fd37-42d9-85ed-8456fae63af0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:57.220276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8830a80a-fd37-42d9-85ed-8456fae63af0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:57.222779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8830a80a-fd37-42d9-85ed-8456fae63af0 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:57.234217 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1287/2554] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:23:57.425266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:57.528353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2170ad9-3351-4c09-8931-aa9763b0d279 service placement] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:57.533610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:57.534582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:57.535057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:57.535396 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] PATH_INFO: `/v3/users/7c6abe27ceac49a1abf3dcc083c5006d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:57.536602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c6abe27ceac49a1abf3dcc083c5006d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:57.539045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:57.625104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c6abe27ceac49a1abf3dcc083c5006d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:57.626217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c6abe27ceac49a1abf3dcc083c5006d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:57.648319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c6abe27ceac49a1abf3dcc083c5006d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:57.649994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c6abe27ceac49a1abf3dcc083c5006d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:57.659530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c6abe27ceac49a1abf3dcc083c5006d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:57.695940 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1268/2555] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:23:57 2022] DELETE /identity/v3/users/7c6abe27ceac49a1abf3dcc083c5006d => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:23:57.701196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:57.701802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:57.702066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:57.727762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:57.729569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:57.816634 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1288/2556] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:57.827542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:57.829008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:57.894235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b320ed49-2eea-47aa-bec7-16868d7d2b86 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:57.900720 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc45c23-9989-4903-83da-12a74e0a75fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:57.901921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc45c23-9989-4903-83da-12a74e0a75fd service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:57.902507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc45c23-9989-4903-83da-12a74e0a75fd service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:57.903008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc45c23-9989-4903-83da-12a74e0a75fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:57.917874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc45c23-9989-4903-83da-12a74e0a75fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:57.919361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc45c23-9989-4903-83da-12a74e0a75fd service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:57.930387 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1269/2557] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:23:58.445332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:58.446522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:58.461808 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea963408-3013-4747-b6f4-0c1386cc7ace None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:58.468131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:58.480253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:58.480913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.481219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.481495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] PATH_INFO: `/v3/projects/89afa93271f944eb9324505f73422d54` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.482711 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] RBAC: Authorizing `identity:delete_project(project_id=89afa93271f944eb9324505f73422d54)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:58.486334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:58.555775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89afa93271f944eb9324505f73422d54', 'request_id': 'req-2943be29-6545-46c4-86d1-59a018fd7abb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:58.564136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89afa93271f944eb9324505f73422d54', 'request_id': 'req-2943be29-6545-46c4-86d1-59a018fd7abb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:58.572963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89afa93271f944eb9324505f73422d54', 'request_id': 'req-2943be29-6545-46c4-86d1-59a018fd7abb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:58.584060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] The token cache is being invalidate because project 89afa93271f944eb9324505f73422d54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:23:58.584359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:58.585050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2943be29-6545-46c4-86d1-59a018fd7abb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:23:58.587806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1289/2558] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:23:58 2022] DELETE /identity/v3/projects/89afa93271f944eb9324505f73422d54 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:58.592637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53252c9e-aacf-40a0-a576-d86c80e66943 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.593156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53252c9e-aacf-40a0-a576-d86c80e66943 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.593441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53252c9e-aacf-40a0-a576-d86c80e66943 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.608766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:23:58.617176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-53252c9e-aacf-40a0-a576-d86c80e66943 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:58.619047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-53252c9e-aacf-40a0-a576-d86c80e66943 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:58.687118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1270/2559] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:58.695558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86949fb-b843-443e-a800-394be72a35fd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.695931 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86949fb-b843-443e-a800-394be72a35fd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.696226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86949fb-b843-443e-a800-394be72a35fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.716803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c86949fb-b843-443e-a800-394be72a35fd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:58.718112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c86949fb-b843-443e-a800-394be72a35fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:58.726114 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1290/2560] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:23:58.730500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53252c9e-aacf-40a0-a576-d86c80e66943 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:58.740036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53252c9e-aacf-40a0-a576-d86c80e66943 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:58.744956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:58.745547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.745831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.746103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.746749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:23:58.747053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:58.748679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:58.760968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1271/2561] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:23:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:23:58.765447 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86949fb-b843-443e-a800-394be72a35fd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:58.776795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86949fb-b843-443e-a800-394be72a35fd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:58.781442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:58.782070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.782349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.782621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.783249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:58.785174 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:58.814734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1291/2562] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:23:58 2022] POST /identity/v3/projects => generated 381 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:58.818672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:58.835635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa79067-63b0-4b88-b144-ca9ff64d6e5d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:58.890864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:58.891474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.891757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.892033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.892664 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:58.894661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:58.917072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1272/2563] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:23:58 2022] POST /identity/v3/users => generated 291 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:23:58.921327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:58.931974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de73413-9b85-4171-8b98-1119a593232f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:58.936758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:58.937567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.937920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.938257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.938966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:58.940630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:58.951068 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1292/2564] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:23:58.955144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:58.966157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeb271a-fa42-4156-9a16-7a019b26fb4b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:58.970926 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38c3ef6-9b50-4a3b-ba1d-3a71c9df6fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:58.971526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c3ef6-9b50-4a3b-ba1d-3a71c9df6fc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:58.971863 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c3ef6-9b50-4a3b-ba1d-3a71c9df6fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:58.972146 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38c3ef6-9b50-4a3b-ba1d-3a71c9df6fc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:58.972784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38c3ef6-9b50-4a3b-ba1d-3a71c9df6fc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:58.974394 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38c3ef6-9b50-4a3b-ba1d-3a71c9df6fc3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:58.986441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1273/2565] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:23:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:23:58.990737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:59.003227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec503120-565a-4938-b530-d73583c8976c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:59.013070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:59.013734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:59.014017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:59.014290 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] PATH_INFO: `/v3/projects/b8a01a630605461e8facb8fe09cf3db1/users/f78d89d132b845c8be8e38ff06b6a09f/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:59.029708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8a01a630605461e8facb8fe09cf3db1, user_id=f78d89d132b845c8be8e38ff06b6a09f, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:59.032358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:59.045802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1293/2566] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:23:58 2022] PUT /identity/v3/projects/b8a01a630605461e8facb8fe09cf3db1/users/f78d89d132b845c8be8e38ff06b6a09f/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:23:59.056117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513938b1-8ffb-430a-adff-509279efe80f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:59.056401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513938b1-8ffb-430a-adff-509279efe80f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:59.056670 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513938b1-8ffb-430a-adff-509279efe80f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:59.084565 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-513938b1-8ffb-430a-adff-509279efe80f None None] MFA Rules not processed for user `f78d89d132b845c8be8e38ff06b6a09f`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:23:59.086299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-513938b1-8ffb-430a-adff-509279efe80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:59.152455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1274/2567] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:23:59 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:23:59.164017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:23:59.165690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:23:59.224196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f78ab5a-4287-4744-a358-fb67d376eabf admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:23:59.228597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf40b6f-b5c6-462e-bbcc-bf6b21c606fa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:23:59.229277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf40b6f-b5c6-462e-bbcc-bf6b21c606fa service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:23:59.229562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf40b6f-b5c6-462e-bbcc-bf6b21c606fa service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:23:59.229844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf40b6f-b5c6-462e-bbcc-bf6b21c606fa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:23:59.238164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf40b6f-b5c6-462e-bbcc-bf6b21c606fa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:23:59.239595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf40b6f-b5c6-462e-bbcc-bf6b21c606fa service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:23:59.248481 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1294/2568] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:23:59 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:01.860469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513938b1-8ffb-430a-adff-509279efe80f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:01.864989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:01.958121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513938b1-8ffb-430a-adff-509279efe80f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:01.967330 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:01.968370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:01.969687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:01.970248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] PATH_INFO: `/v3/users/f78d89d132b845c8be8e38ff06b6a09f` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:01.976596 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] RBAC: Authorizing `identity:delete_user(user_id=f78d89d132b845c8be8e38ff06b6a09f)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:01.978866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:02.071949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f78d89d132b845c8be8e38ff06b6a09f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.072952 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f78d89d132b845c8be8e38ff06b6a09f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.081495 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f78d89d132b845c8be8e38ff06b6a09f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.082540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f78d89d132b845c8be8e38ff06b6a09f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.092502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f78d89d132b845c8be8e38ff06b6a09f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.111936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1275/2569] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:01 2022] DELETE /identity/v3/users/f78d89d132b845c8be8e38ff06b6a09f => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:02.116586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.117224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.117224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.136110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:02.137407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.158463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:02.211379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1295/2570] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:02.221869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:02.223526 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.288981 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8d26f2-856b-4064-9d3b-a1c8f503032c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:02.296093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b29c1c-3e1e-4a17-b03b-beda26af8890 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:02.296838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b29c1c-3e1e-4a17-b03b-beda26af8890 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.298303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b29c1c-3e1e-4a17-b03b-beda26af8890 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.298303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b29c1c-3e1e-4a17-b03b-beda26af8890 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.309292 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b29c1c-3e1e-4a17-b03b-beda26af8890 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:02.310874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b29c1c-3e1e-4a17-b03b-beda26af8890 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:02.317091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:02.318576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.324051 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1276/2571] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:24:02.332306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93449580-d778-4c94-8c25-ff6091d90e93 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:02.337799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca3d9c-b403-423b-8d75-7cc91a5a84c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:02.338525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca3d9c-b403-423b-8d75-7cc91a5a84c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.338863 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca3d9c-b403-423b-8d75-7cc91a5a84c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.339204 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca3d9c-b403-423b-8d75-7cc91a5a84c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.341074 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-85ca3d9c-b403-423b-8d75-7cc91a5a84c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.351467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ca3d9c-b403-423b-8d75-7cc91a5a84c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:02.353106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ca3d9c-b403-423b-8d75-7cc91a5a84c1 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:02.362569 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1296/2572] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:24:02.503779 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0a37bd-7830-49a9-a734-f6098bc24123 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.504109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0a37bd-7830-49a9-a734-f6098bc24123 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.504447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0a37bd-7830-49a9-a734-f6098bc24123 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.505890 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1277/2573] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:24:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 01 04:24:02.513130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.513664 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.514124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.546510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:02.548090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.617641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1297/2574] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:24:02 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:02.629931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0a37bd-7830-49a9-a734-f6098bc24123 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:02.631704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ea0a37bd-7830-49a9-a734-f6098bc24123 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.661574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:02.750793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0a37bd-7830-49a9-a734-f6098bc24123 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:02.755584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c805706f-47c8-4639-b0bf-6c670141f91c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '170fe0394f3d4472a1380481cfe2cc28', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:02.756824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805706f-47c8-4639-b0bf-6c670141f91c service glance] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.757377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805706f-47c8-4639-b0bf-6c670141f91c service glance] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.757819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c805706f-47c8-4639-b0bf-6c670141f91c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.764064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:02.765516 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.773968 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c805706f-47c8-4639-b0bf-6c670141f91c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:02.775719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c805706f-47c8-4639-b0bf-6c670141f91c service glance] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:02.780153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e2cc0e-b95a-43a8-8346-1a648b6ff3bb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:02.785276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:02.786076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.786478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.786852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] PATH_INFO: `/v3/projects/b8a01a630605461e8facb8fe09cf3db1` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.788214 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8a01a630605461e8facb8fe09cf3db1)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:02.790945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:02.795547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1278/2575] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:24:02.861655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8a01a630605461e8facb8fe09cf3db1', 'request_id': 'req-3f1575af-63e2-42a4-b121-5d16cdc17565'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.872235 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8a01a630605461e8facb8fe09cf3db1', 'request_id': 'req-3f1575af-63e2-42a4-b121-5d16cdc17565'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.878283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83534519-e636-45d0-b46d-cfa8ee3e1567 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.878785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83534519-e636-45d0-b46d-cfa8ee3e1567 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.878902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83534519-e636-45d0-b46d-cfa8ee3e1567 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.880220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1279/2576] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:24:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:24:02.881059 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8a01a630605461e8facb8fe09cf3db1', 'request_id': 'req-3f1575af-63e2-42a4-b121-5d16cdc17565'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.885901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.886182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.886466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.896283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] The token cache is being invalidate because project b8a01a630605461e8facb8fe09cf3db1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:02.896576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.897971 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3f1575af-63e2-42a4-b121-5d16cdc17565 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:02.899976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1298/2577] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:02 2022] DELETE /identity/v3/projects/b8a01a630605461e8facb8fe09cf3db1 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:24:02.905292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd041-8d6d-45ce-9b91-918a3ca141d0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:02.905763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd041-8d6d-45ce-9b91-918a3ca141d0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:02.906181 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011bd041-8d6d-45ce-9b91-918a3ca141d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:02.910669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] MFA Rules not processed for user `16aab142c0a7445483caf19178d56062`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:02.912080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.935438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-011bd041-8d6d-45ce-9b91-918a3ca141d0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:02.939194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-011bd041-8d6d-45ce-9b91-918a3ca141d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:02.981783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:02.984226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1280/2578] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:24:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:03.002427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.006401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:03.018501 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0971884-ea4e-43ea-8cab-5a13f50e48bc None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.022579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.023194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.023473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.023748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.028310 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1299/2579] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:03.031176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.032674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.036838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e33a519-5ff6-4622-9bc7-c6f2385a470a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.037177 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e33a519-5ff6-4622-9bc7-c6f2385a470a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.038348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e33a519-5ff6-4622-9bc7-c6f2385a470a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.041843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1281/2580] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:03.061926 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0e33a519-5ff6-4622-9bc7-c6f2385a470a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:03.063135 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e33a519-5ff6-4622-9bc7-c6f2385a470a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:03.072799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1300/2581] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:03.076629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.092460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03be2e5a-0934-4953-ba6a-e5d3978a8d43 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.097388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220687b5-e9b4-4bfa-911e-70dec79364bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.098127 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220687b5-e9b4-4bfa-911e-70dec79364bc admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.098416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220687b5-e9b4-4bfa-911e-70dec79364bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.098695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220687b5-e9b4-4bfa-911e-70dec79364bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.099379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220687b5-e9b4-4bfa-911e-70dec79364bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:24:03.099686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220687b5-e9b4-4bfa-911e-70dec79364bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.101891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220687b5-e9b4-4bfa-911e-70dec79364bc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.118901 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1282/2582] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:24:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:24:03.120397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e33a519-5ff6-4622-9bc7-c6f2385a470a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.130696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e33a519-5ff6-4622-9bc7-c6f2385a470a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.135966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.136652 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.137056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.137535 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.138238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.140206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.145535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf0736-ecba-4eeb-b04d-4f51a06012e2 None None] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.146801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf0736-ecba-4eeb-b04d-4f51a06012e2 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.147151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf0736-ecba-4eeb-b04d-4f51a06012e2 None None] PATH_INFO: `/` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.149288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1283/2583] 10.222.0.26 () {54 vars in 921 bytes} [Sat Oct 1 04:24:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 01 04:24:03.155048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07812d34-fbe7-4e98-af76-d0ae20ec9c8d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.155419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07812d34-fbe7-4e98-af76-d0ae20ec9c8d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.155851 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07812d34-fbe7-4e98-af76-d0ae20ec9c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.177071 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1301/2584] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:03 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:03.178356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.185662 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-07812d34-fbe7-4e98-af76-d0ae20ec9c8d None None] MFA Rules not processed for user `901b9bb47c2f40c2b1b4cfe7886a1167`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:03.186543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-07812d34-fbe7-4e98-af76-d0ae20ec9c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:03.253891 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff45a8c-ada5-49f3-9d21-448881000867 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.259620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.260278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.260490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.260756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.261415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.263323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.270708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1284/2585] 10.222.0.26 () {58 vars in 1018 bytes} [Sat Oct 1 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:03.282250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07812d34-fbe7-4e98-af76-d0ae20ec9c8d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.291238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1302/2586] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:03 2022] POST /identity/v3/users => generated 288 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:24:03.293781 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.304758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8c1167-a6e3-476f-adc0-41ebdd6d5fe5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.309862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.310467 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.310820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.311027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.311672 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.313390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.325380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1303/2587] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:03.329160 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.332280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:03.339622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de768b51-f079-4353-985c-937a6f1e9a22 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.344370 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.344563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07812d34-fbe7-4e98-af76-d0ae20ec9c8d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.345008 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.345292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.345567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.346200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.347780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.349350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4640e01-b313-42e4-a423-6b5f4520ecef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.350186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4640e01-b313-42e4-a423-6b5f4520ecef service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.351467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4640e01-b313-42e4-a423-6b5f4520ecef service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.351870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4640e01-b313-42e4-a423-6b5f4520ecef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.359474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1304/2588] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:24:03.363645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.365484 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4640e01-b313-42e4-a423-6b5f4520ecef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.367665 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4640e01-b313-42e4-a423-6b5f4520ecef service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.374800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2d197d-0f27-4af0-b5b3-47f73a36ae30 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.380172 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.380894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.381153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.381429 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] PATH_INFO: `/v3/projects/606b81358ba64ffd83b8ef1cf810e52c/users/b2b4e0b75ed340e1b95e7af948828b32/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.381951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1285/2589] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:24:03.397250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=606b81358ba64ffd83b8ef1cf810e52c, user_id=b2b4e0b75ed340e1b95e7af948828b32, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.400091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.419269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1305/2590] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:03 2022] PUT /identity/v3/projects/606b81358ba64ffd83b8ef1cf810e52c/users/b2b4e0b75ed340e1b95e7af948828b32/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:24:03.429984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b3f285-eefa-47e3-addd-ff7e1950c16a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.430547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b3f285-eefa-47e3-addd-ff7e1950c16a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.431048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b3f285-eefa-47e3-addd-ff7e1950c16a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.477581 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-05b3f285-eefa-47e3-addd-ff7e1950c16a None None] MFA Rules not processed for user `b2b4e0b75ed340e1b95e7af948828b32`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:03.479055 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-05b3f285-eefa-47e3-addd-ff7e1950c16a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:03.582830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1286/2591] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:03.594442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:03.597129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:03.663030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:03.678235 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f0b890-9f1a-4eb2-92c9-2b9cf16ffa85 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:03.681976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:03.682576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:03.682850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:03.683122 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:03.692523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:03.693993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:03.704999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1306/2592] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:03 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:06.652095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b2f61a-d108-4c41-bc98-974a51c47e79 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:06.652582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b2f61a-d108-4c41-bc98-974a51c47e79 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:06.652702 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b2f61a-d108-4c41-bc98-974a51c47e79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:06.672385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-01b2f61a-d108-4c41-bc98-974a51c47e79 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:06.673616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-01b2f61a-d108-4c41-bc98-974a51c47e79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:06.742075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1287/2593] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:24:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:06.770137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:06.771820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:06.868230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50f604-e62f-4813-91af-bdf7550ae5f9 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:06.873466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2174f82-782e-4c7d-a729-c39a2da51e52 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:06.874388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2174f82-782e-4c7d-a729-c39a2da51e52 service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:06.874807 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2174f82-782e-4c7d-a729-c39a2da51e52 service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:06.875288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2174f82-782e-4c7d-a729-c39a2da51e52 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:06.885912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2174f82-782e-4c7d-a729-c39a2da51e52 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:06.887725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2174f82-782e-4c7d-a729-c39a2da51e52 service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:06.905293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1307/2594] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:24:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:24:14.632179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b2f61a-d108-4c41-bc98-974a51c47e79 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:14.700285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b2f61a-d108-4c41-bc98-974a51c47e79 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:14.705288 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:14.705881 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:14.706158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:14.706436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] PATH_INFO: `/v3/users/b2b4e0b75ed340e1b95e7af948828b32` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:14.707612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2b4e0b75ed340e1b95e7af948828b32)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:14.711419 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:14.745648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:14.771417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2b4e0b75ed340e1b95e7af948828b32'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:14.772803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2b4e0b75ed340e1b95e7af948828b32'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:14.780204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2b4e0b75ed340e1b95e7af948828b32'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:14.780948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2b4e0b75ed340e1b95e7af948828b32'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:14.790792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2b4e0b75ed340e1b95e7af948828b32'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:14.817246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1288/2595] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:14 2022] DELETE /identity/v3/users/b2b4e0b75ed340e1b95e7af948828b32 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:14.822816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:14.823332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:14.823715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:14.845056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:14.846293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:14.915463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1308/2596] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:14.926087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:14.927498 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:15.006990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c093c66-f7c3-4f50-83da-05974d3ee263 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:15.011982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61832a2-1169-4bac-a9ce-4f62a26b834a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:15.012911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61832a2-1169-4bac-a9ce-4f62a26b834a service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:15.013398 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61832a2-1169-4bac-a9ce-4f62a26b834a service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:15.013704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61832a2-1169-4bac-a9ce-4f62a26b834a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:15.026620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61832a2-1169-4bac-a9ce-4f62a26b834a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:15.028982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61832a2-1169-4bac-a9ce-4f62a26b834a service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:15.045203 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1289/2597] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:15.446225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:15.448647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:15.468243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f53295-fcab-417b-90cb-62e69a0cff27 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:15.475511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:15.476377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:15.476684 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:15.477070 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] PATH_INFO: `/v3/projects/606b81358ba64ffd83b8ef1cf810e52c` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:15.479319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] RBAC: Authorizing `identity:delete_project(project_id=606b81358ba64ffd83b8ef1cf810e52c)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:15.481559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:15.555228 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '606b81358ba64ffd83b8ef1cf810e52c', 'request_id': 'req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:15.562946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '606b81358ba64ffd83b8ef1cf810e52c', 'request_id': 'req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:15.570597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '606b81358ba64ffd83b8ef1cf810e52c', 'request_id': 'req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:15.577909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:15.582693 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] The token cache is being invalidate because project 606b81358ba64ffd83b8ef1cf810e52c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:15.583011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:15.583822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2b18bf27-3a4d-4633-99b4-a9c1f5fa5771 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:15.586658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1309/2598] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:15 2022] DELETE /identity/v3/projects/606b81358ba64ffd83b8ef1cf810e52c => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:15.593902 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6faf56c-8697-47b0-a57a-660120908287 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:15.594185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6faf56c-8697-47b0-a57a-660120908287 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:15.594462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6faf56c-8697-47b0-a57a-660120908287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:15.625594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-f6faf56c-8697-47b0-a57a-660120908287 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:15.626956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f6faf56c-8697-47b0-a57a-660120908287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:15.763855 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1290/2599] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:15.766358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc23e2fa-990e-441f-a362-0cfc63c1eb25 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:15.766649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc23e2fa-990e-441f-a362-0cfc63c1eb25 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:15.766932 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc23e2fa-990e-441f-a362-0cfc63c1eb25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:15.793901 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-dc23e2fa-990e-441f-a362-0cfc63c1eb25 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:15.795395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-dc23e2fa-990e-441f-a362-0cfc63c1eb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:15.807658 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1310/2600] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:15.811960 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6faf56c-8697-47b0-a57a-660120908287 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:15.827911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6faf56c-8697-47b0-a57a-660120908287 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:15.833522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:15.834137 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:15.834534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:15.834809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:15.835464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:24:15.835766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:15.844132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:15.866135 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1291/2601] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:24:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:24:15.870333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc23e2fa-990e-441f-a362-0cfc63c1eb25 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:15.880148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc23e2fa-990e-441f-a362-0cfc63c1eb25 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:15.886547 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:15.887166 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:15.887451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:15.887852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:15.888656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:15.890999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:15.931094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1311/2602] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:15 2022] POST /identity/v3/projects => generated 371 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:24:15.935433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:15.947462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d932353a-ee04-42a7-8410-cfe3c7bb3ca5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:15.976534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:16.013663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:16.014647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:16.014898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:16.015204 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:16.016151 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:16.018144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:16.041157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1292/2603] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:15 2022] POST /identity/v3/users => generated 286 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:16.046364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:16.057854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fc9da3-92cc-4435-924e-bcebb57bc3b2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:16.070817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:16.071703 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:16.072376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:16.072767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:16.074434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:16.076377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:16.092409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1312/2604] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:16 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:16.096566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:16.106777 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ebc063-c240-4a4c-9dc5-8af414a094d2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:16.111253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb420c0-861e-4f56-8b8e-835f4811c9a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:16.111840 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb420c0-861e-4f56-8b8e-835f4811c9a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:16.112116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb420c0-861e-4f56-8b8e-835f4811c9a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:16.112386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb420c0-861e-4f56-8b8e-835f4811c9a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:16.113064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb420c0-861e-4f56-8b8e-835f4811c9a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:16.114649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb420c0-861e-4f56-8b8e-835f4811c9a3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:16.125848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1293/2605] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:16.132878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:16.143841 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae6b70-050a-490e-9fd8-87a433a40075 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:16.149719 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:16.150394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:16.150656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:16.150927 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] PATH_INFO: `/v3/projects/1eda12202e6f439b96a87a81d7531503/users/944a3ab108184d5ab8784e21805f9873/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:16.166296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] RBAC: Authorizing `identity:create_grant(project_id=1eda12202e6f439b96a87a81d7531503, user_id=944a3ab108184d5ab8784e21805f9873, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:16.169397 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:16.196468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1313/2606] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:16 2022] PUT /identity/v3/projects/1eda12202e6f439b96a87a81d7531503/users/944a3ab108184d5ab8784e21805f9873/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:16.201632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564b8a39-dd38-49df-8099-abd1f5a82250 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:16.201914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564b8a39-dd38-49df-8099-abd1f5a82250 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:16.202185 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564b8a39-dd38-49df-8099-abd1f5a82250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:16.228329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-564b8a39-dd38-49df-8099-abd1f5a82250 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:16.229947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-564b8a39-dd38-49df-8099-abd1f5a82250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:16.326891 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1294/2607] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:16.332478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:16.335239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:16.423850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8c8615-a4d6-4c2b-abdf-4abcecfff66a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:16.433502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:16.434885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:16.434885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:16.435231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:16.451941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:16.454237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:16.468493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1314/2608] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:21.350829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ab8f55-e2c4-408e-897a-4b95f81eb5f9 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:21.351378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ab8f55-e2c4-408e-897a-4b95f81eb5f9 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:21.351378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ab8f55-e2c4-408e-897a-4b95f81eb5f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:21.387466 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-43ab8f55-e2c4-408e-897a-4b95f81eb5f9 None None] MFA Rules not processed for user `944a3ab108184d5ab8784e21805f9873`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:21.388913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-43ab8f55-e2c4-408e-897a-4b95f81eb5f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:21.479557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1295/2609] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:21 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:21.480792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:21.492899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33761a60-d231-44cf-9528-db911e24378a service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:21.498438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:21.499180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:21.499487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:21.499837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:21.500501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:21.502474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:21.539904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1315/2610] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:21 2022] POST /identity/v3/projects => generated 373 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:21.544256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ab8f55-e2c4-408e-897a-4b95f81eb5f9 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:21.558120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ab8f55-e2c4-408e-897a-4b95f81eb5f9 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:21.647601 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:21.681474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:21.682509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:21.682798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:21.683326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:21.683933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:21.686395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:21.716515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1296/2611] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:21 2022] POST /identity/v3/users => generated 293 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:21.720501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:21.746165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75881e4-2d24-4029-ba43-d88299795e8e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:21.751646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:21.752528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:21.752995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:21.753855 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:21.754986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:21.757501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:21.773552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1316/2612] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:21 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:21.777780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:21.789671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3eda064-8262-4bf8-9c62-3a2c8eddaf21 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:21.799350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:21.800285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:21.800876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:21.801430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] PATH_INFO: `/v3/projects/2dccf74ffdb5456e88c122f02cb84f0f/users/edf10e00650444f6b379d694d80f771d/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:21.820776 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dccf74ffdb5456e88c122f02cb84f0f, user_id=edf10e00650444f6b379d694d80f771d, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:21.823921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:21.844737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1297/2613] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:21 2022] PUT /identity/v3/projects/2dccf74ffdb5456e88c122f02cb84f0f/users/edf10e00650444f6b379d694d80f771d/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:21.848668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:21.861348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cab8077-466a-4525-81ec-90fa0776b6ce admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:22.014651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029bcd0-d976-4ef4-9746-b2f7646ab9de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:22.016018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029bcd0-d976-4ef4-9746-b2f7646ab9de admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:22.016465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029bcd0-d976-4ef4-9746-b2f7646ab9de admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:22.017194 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029bcd0-d976-4ef4-9746-b2f7646ab9de admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:22.018030 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8029bcd0-d976-4ef4-9746-b2f7646ab9de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:22.019936 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8029bcd0-d976-4ef4-9746-b2f7646ab9de admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:22.032407 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1317/2614] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:21 2022] GET /identity/v3/roles => generated 1420 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:24:22.042558 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:22.057297 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff770a39-066f-48b4-ba38-c9a84bde8797 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:22.062770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:22.063460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:22.063741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:22.064019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] PATH_INFO: `/v3/projects/2dccf74ffdb5456e88c122f02cb84f0f/users/edf10e00650444f6b379d694d80f771d/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:22.073869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dccf74ffdb5456e88c122f02cb84f0f, user_id=edf10e00650444f6b379d694d80f771d, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:22.077041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:22.096559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1298/2615] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:22 2022] PUT /identity/v3/projects/2dccf74ffdb5456e88c122f02cb84f0f/users/edf10e00650444f6b379d694d80f771d/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:23.883485 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:23.884018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:23.884018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:23.899003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:23.922563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:23.924138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:24.057669 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:24.062005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:24.078816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1318/2616] 10.222.0.26 () {58 vars in 967 bytes} [Sat Oct 1 04:24:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:24.083292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:24.088272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:24.266748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91085fae-2d14-4076-9ee5-d6357434480d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:24.286047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:24.287876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:24.288170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:24.288454 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:24.294730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:24.316408 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d36be5-bcdb-498c-9d76-1636e7cba7f8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:24.332961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70ceaa8-45ed-42cf-a8c1-096a85efe67c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:24.337365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ceaa8-45ed-42cf-a8c1-096a85efe67c service swift] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:24.338249 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ceaa8-45ed-42cf-a8c1-096a85efe67c service swift] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:24.340559 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ceaa8-45ed-42cf-a8c1-096a85efe67c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:24.360149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70ceaa8-45ed-42cf-a8c1-096a85efe67c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:24.362635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70ceaa8-45ed-42cf-a8c1-096a85efe67c service swift] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:24.386921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1319/2617] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:24:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:24:24.436991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:24.439908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:24.464624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1299/2618] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:28.897974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:28.898492 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:28.898578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:28.971013 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] MFA Rules not processed for user `edf10e00650444f6b379d694d80f771d`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:28.972759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:29.079162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1320/2619] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:28 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:29.093884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:29.098001 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:29.160296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:29.195697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e4f91-9f40-48b4-9edf-5a55b06c547c service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:29.201828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:29.202362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:29.202642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:29.202918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:29.284671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:29.287363 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:29.301782 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1300/2620] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:29 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:29.563475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:29.563475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:29.589741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:29.633118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:29.635241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:29.651221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae3356a-4318-42f7-bae6-f9255e9d81c8 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:29.658126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8386c304-bd9f-4393-99fe-6b138801c586 service nova] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:29.663696 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:29.665723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:29.665723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:29.665723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:29.668918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:29.669921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:29.670796 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:29.672358 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:29.672747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] PATH_INFO: `/v3/users/3caf1cc366434d418c474d7a1879faa7` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:29.680121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3caf1cc366434d418c474d7a1879faa7)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:29.683075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:29.707580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:29.709870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:29.726784 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1321/2621] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:29.791848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3caf1cc366434d418c474d7a1879faa7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:29.793029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3caf1cc366434d418c474d7a1879faa7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:29.804150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3caf1cc366434d418c474d7a1879faa7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:29.804898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3caf1cc366434d418c474d7a1879faa7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:29.815071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6d4fcb92-9ef5-49cd-aa03-b392199307d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3caf1cc366434d418c474d7a1879faa7'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:29.839408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1301/2622] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:29 2022] DELETE /identity/v3/users/3caf1cc366434d418c474d7a1879faa7 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:24:30.067093 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.080324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3142ad-6fc6-450a-a9b6-c99c69ac1ca2 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:30.147120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:30.147799 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.148075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.148344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] PATH_INFO: `/v3/projects/16cd7bcb315d4dc58f73a9918adb57e2` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.149486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16cd7bcb315d4dc58f73a9918adb57e2)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:30.151544 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:30.200895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16cd7bcb315d4dc58f73a9918adb57e2', 'request_id': 'req-db904bdb-3064-4be2-9851-48ddd6157cd4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:30.212227 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16cd7bcb315d4dc58f73a9918adb57e2', 'request_id': 'req-db904bdb-3064-4be2-9851-48ddd6157cd4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:30.226725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16cd7bcb315d4dc58f73a9918adb57e2', 'request_id': 'req-db904bdb-3064-4be2-9851-48ddd6157cd4'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:30.254936 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] The token cache is being invalidate because project 16cd7bcb315d4dc58f73a9918adb57e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:30.255340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:30.256229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-db904bdb-3064-4be2-9851-48ddd6157cd4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:30.258508 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1322/2623] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:30 2022] DELETE /identity/v3/projects/16cd7bcb315d4dc58f73a9918adb57e2 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:30.263191 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb46df8-978c-4c34-b9d0-58a6ef0a3ee8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.263705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb46df8-978c-4c34-b9d0-58a6ef0a3ee8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.263848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb46df8-978c-4c34-b9d0-58a6ef0a3ee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.291537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-feb46df8-978c-4c34-b9d0-58a6ef0a3ee8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:30.293142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-feb46df8-978c-4c34-b9d0-58a6ef0a3ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:30.344907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:30.373241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1302/2624] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:30.381502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003fbb4e-8035-400d-86d8-870843b53f25 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.382113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003fbb4e-8035-400d-86d8-870843b53f25 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.382619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003fbb4e-8035-400d-86d8-870843b53f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.410590 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-003fbb4e-8035-400d-86d8-870843b53f25 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:30.412963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-003fbb4e-8035-400d-86d8-870843b53f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:30.434659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1323/2625] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:30.436588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb46df8-978c-4c34-b9d0-58a6ef0a3ee8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.448266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb46df8-978c-4c34-b9d0-58a6ef0a3ee8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:30.452117 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:30.452718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.453035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.453306 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.453964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:24:30.454271 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:30.456585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:30.473973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1303/2626] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:24:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:24:30.478738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003fbb4e-8035-400d-86d8-870843b53f25 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.488218 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003fbb4e-8035-400d-86d8-870843b53f25 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:30.496642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:30.497307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.498034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.498322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.498984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:30.505083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:30.549411 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.557165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1324/2627] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:30 2022] POST /identity/v3/projects => generated 383 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:30.563063 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7216df-19c0-4601-9881-9f0bfc94ddb8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:30.641955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:30.642802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.643184 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.643582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.644449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:30.647174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:30.679846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1304/2628] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:30 2022] POST /identity/v3/users => generated 292 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:30.684118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.695763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795170d5-2530-41c9-9a04-e93c5b598d39 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:30.701737 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:30.702521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.702789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.703133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.703976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:30.706046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:30.718657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1325/2629] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:30 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:30.722770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.733845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2457d6-0b3f-4091-a979-1c3f165c176b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:30.740352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d38cd8-3581-41ce-bf12-ed77b1bb07b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:30.740955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d38cd8-3581-41ce-bf12-ed77b1bb07b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.741235 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d38cd8-3581-41ce-bf12-ed77b1bb07b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.741824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d38cd8-3581-41ce-bf12-ed77b1bb07b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.742142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d38cd8-3581-41ce-bf12-ed77b1bb07b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:30.743759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d38cd8-3581-41ce-bf12-ed77b1bb07b9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:30.757236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1305/2630] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:30 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:30.761531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.767061 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:30.780958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a33a2-5c94-4955-aee5-968f1bd390ed admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:30.789777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998ca44f-3f6b-473e-9ccb-c0fda8b27d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:30.789777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998ca44f-3f6b-473e-9ccb-c0fda8b27d1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.789777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998ca44f-3f6b-473e-9ccb-c0fda8b27d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.789777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998ca44f-3f6b-473e-9ccb-c0fda8b27d1b admin admin] PATH_INFO: `/v3/projects/da8e7aa8a88d40dfbbecf0bc3f51ac5d/users/4a73940f07a049b2b59e4ac333004d95/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.811893 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998ca44f-3f6b-473e-9ccb-c0fda8b27d1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=da8e7aa8a88d40dfbbecf0bc3f51ac5d, user_id=4a73940f07a049b2b59e4ac333004d95, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:30.814980 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998ca44f-3f6b-473e-9ccb-c0fda8b27d1b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:30.832973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1326/2631] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:30 2022] PUT /identity/v3/projects/da8e7aa8a88d40dfbbecf0bc3f51ac5d/users/4a73940f07a049b2b59e4ac333004d95/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:30.844725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af72aa-1fb4-4185-a2a8-58d30b506a29 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.845100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af72aa-1fb4-4185-a2a8-58d30b506a29 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.845323 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af72aa-1fb4-4185-a2a8-58d30b506a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.873321 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:30.873694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:30.873870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:30.886668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b7af72aa-1fb4-4185-a2a8-58d30b506a29 None None] MFA Rules not processed for user `4a73940f07a049b2b59e4ac333004d95`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:30.888355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b7af72aa-1fb4-4185-a2a8-58d30b506a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:30.894642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] MFA Rules not processed for user `944a3ab108184d5ab8784e21805f9873`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:30.896304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:30.957056 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1327/2632] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:30 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:30.962108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1306/2633] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:30.972504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:30.975406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:31.058941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6af865f-9a18-49a1-87d7-f7551583ed30 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:31.065332 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0211f100-9bd9-4251-909c-c6382bb7ba58 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:31.066753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211f100-9bd9-4251-909c-c6382bb7ba58 service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:31.067036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211f100-9bd9-4251-909c-c6382bb7ba58 service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:31.067309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211f100-9bd9-4251-909c-c6382bb7ba58 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:31.077547 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0211f100-9bd9-4251-909c-c6382bb7ba58 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:31.079349 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0211f100-9bd9-4251-909c-c6382bb7ba58 service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:31.089155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1328/2634] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:30 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:39.545212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7af72aa-1fb4-4185-a2a8-58d30b506a29 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:39.588183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:39.608385 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7af72aa-1fb4-4185-a2a8-58d30b506a29 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:39.612830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:39.613487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:39.613759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:39.614030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] PATH_INFO: `/v3/users/4a73940f07a049b2b59e4ac333004d95` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:39.615134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a73940f07a049b2b59e4ac333004d95)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:39.617205 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:39.666534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a73940f07a049b2b59e4ac333004d95'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:39.667133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a73940f07a049b2b59e4ac333004d95'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:39.674717 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a73940f07a049b2b59e4ac333004d95'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:39.675427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a73940f07a049b2b59e4ac333004d95'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:39.682604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a73940f07a049b2b59e4ac333004d95'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:39.700160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1307/2635] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:39 2022] DELETE /identity/v3/users/4a73940f07a049b2b59e4ac333004d95 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:24:39.704780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:39.705308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:39.705430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:39.727011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:39.728213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:39.787199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1329/2636] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:39.799539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:39.800144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:39.856490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74f15b3-2cb7-485f-a34f-ccbb1e07d20d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:39.860088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15ebeba-c2af-4e27-996d-abb5a1b69f93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:39.860690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15ebeba-c2af-4e27-996d-abb5a1b69f93 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:39.860987 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15ebeba-c2af-4e27-996d-abb5a1b69f93 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:39.861259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15ebeba-c2af-4e27-996d-abb5a1b69f93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:39.870604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15ebeba-c2af-4e27-996d-abb5a1b69f93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:39.872024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15ebeba-c2af-4e27-996d-abb5a1b69f93 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:39.883766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1308/2637] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:40.168912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.170418 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:40.184168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954431ff-b411-4428-bd21-943cb93b8b3d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.188853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.189523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.189822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.190096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] PATH_INFO: `/v3/projects/da8e7aa8a88d40dfbbecf0bc3f51ac5d` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.191256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] RBAC: Authorizing `identity:delete_project(project_id=da8e7aa8a88d40dfbbecf0bc3f51ac5d)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.193193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.196880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:40.254346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da8e7aa8a88d40dfbbecf0bc3f51ac5d', 'request_id': 'req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:40.262583 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da8e7aa8a88d40dfbbecf0bc3f51ac5d', 'request_id': 'req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:40.269207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da8e7aa8a88d40dfbbecf0bc3f51ac5d', 'request_id': 'req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:40.280256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] The token cache is being invalidate because project da8e7aa8a88d40dfbbecf0bc3f51ac5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:40.280543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:40.281188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bc5fd2e7-2081-497a-9d4b-da7cd0f59edb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:40.282659 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1330/2638] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:40 2022] DELETE /identity/v3/projects/da8e7aa8a88d40dfbbecf0bc3f51ac5d => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:40.287450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5efad9a-ff05-4266-bb42-745d5ce57809 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.287741 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5efad9a-ff05-4266-bb42-745d5ce57809 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.288038 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5efad9a-ff05-4266-bb42-745d5ce57809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.308132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e5efad9a-ff05-4266-bb42-745d5ce57809 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:40.309380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e5efad9a-ff05-4266-bb42-745d5ce57809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:40.368449 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1309/2639] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:40.376460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0af6f3-0edf-4be3-89e4-259f4f0d9b3c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.376745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0af6f3-0edf-4be3-89e4-259f4f0d9b3c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.377049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0af6f3-0edf-4be3-89e4-259f4f0d9b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.398265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3e0af6f3-0edf-4be3-89e4-259f4f0d9b3c None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:40.399691 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3e0af6f3-0edf-4be3-89e4-259f4f0d9b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:40.407858 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1331/2640] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:40.412054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5efad9a-ff05-4266-bb42-745d5ce57809 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.422120 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5efad9a-ff05-4266-bb42-745d5ce57809 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.428219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.429393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.429676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.429947 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.430606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:24:40.430906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.432624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.434197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:40.447158 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1310/2641] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:24:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:24:40.451369 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0af6f3-0edf-4be3-89e4-259f4f0d9b3c None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.460507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0af6f3-0edf-4be3-89e4-259f4f0d9b3c None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.466945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.467562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.467838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.468110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.468753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.470668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.499053 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1332/2642] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:40 2022] POST /identity/v3/projects => generated 405 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:40.504030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.516833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd13f552-53b7-41ab-908f-cc8b437b23f8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.577520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.578207 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.578392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.578663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.579307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.581240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.604875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1311/2643] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:40 2022] POST /identity/v3/users => generated 303 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:24:40.609520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.620565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf9a65-b25b-4c30-8779-2ed90f44685c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.625422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.626058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.626341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.626716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.627259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.628859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.639850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1333/2644] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:24:40.643858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.655008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd5b22a-346a-46c9-9154-af4f70f2112f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.659764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1764ad06-1234-435f-8642-b4f9c0e52a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.660393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1764ad06-1234-435f-8642-b4f9c0e52a2a admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.660678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1764ad06-1234-435f-8642-b4f9c0e52a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.660982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1764ad06-1234-435f-8642-b4f9c0e52a2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.661663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1764ad06-1234-435f-8642-b4f9c0e52a2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.663317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1764ad06-1234-435f-8642-b4f9c0e52a2a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.674047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1312/2645] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:40.678272 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.688698 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba38c5c-6ea1-4af6-8fa9-e9b94c316544 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.693497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.695065 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.695531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.695810 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] PATH_INFO: `/v3/projects/c159165317bc41b38f7155d638072688/users/fe362ac84d4d46778ab386d1aa9b9a4a/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.711776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c159165317bc41b38f7155d638072688, user_id=fe362ac84d4d46778ab386d1aa9b9a4a, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.714489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.728131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1334/2646] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:40 2022] PUT /identity/v3/projects/c159165317bc41b38f7155d638072688/users/fe362ac84d4d46778ab386d1aa9b9a4a/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:40.733405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb2f35-0eba-42bc-b576-0ca50bc63a7e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.733722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb2f35-0eba-42bc-b576-0ca50bc63a7e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.734001 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb2f35-0eba-42bc-b576-0ca50bc63a7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.754291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-95eb2f35-0eba-42bc-b576-0ca50bc63a7e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:40.755499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-95eb2f35-0eba-42bc-b576-0ca50bc63a7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:40.815243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1313/2647] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:40.825721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:40.827900 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:40.892299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db749ab5-6be7-4596-ae77-4b75b141373b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:40.896275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:40.896878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:40.897730 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:40.898052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:40.907577 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:40.909038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:40.919392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1335/2648] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:41.555594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c9328f-602e-474c-a98e-a7b4a90e03aa None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:41.555956 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c9328f-602e-474c-a98e-a7b4a90e03aa None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:41.556260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c9328f-602e-474c-a98e-a7b4a90e03aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:41.588614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-a6c9328f-602e-474c-a98e-a7b4a90e03aa None None] MFA Rules not processed for user `fe362ac84d4d46778ab386d1aa9b9a4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:41.589842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a6c9328f-602e-474c-a98e-a7b4a90e03aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:41.663257 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1314/2649] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:41 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:41.667806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:41.677578 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0589e2e-2fc1-430c-a353-d61c21ef5124 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:41.683280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:41.683887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:41.684172 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:41.684450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:41.685132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:41.687102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:41.720265 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1336/2650] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:41 2022] POST /identity/v3/projects => generated 403 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:41.722182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c9328f-602e-474c-a98e-a7b4a90e03aa None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:41.727833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:41.735118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c9328f-602e-474c-a98e-a7b4a90e03aa None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:41.796281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:41.796887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:41.797200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:41.797471 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:41.798109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:41.799989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:41.821984 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1315/2651] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:41 2022] POST /identity/v3/users => generated 308 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:41.826212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:41.836905 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc5f356-f04b-4ec5-91a7-c81ccd3fb7e4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:41.841490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:41.842317 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:41.842488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:41.842817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:41.843512 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:41.845384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:41.847436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:41.857978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1337/2652] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:24:41.862767 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:41.874028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e025f4ea-9a81-43f3-a8c7-15d96a6226d2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:41.878660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:41.879307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:41.879588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:41.879861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] PATH_INFO: `/v3/projects/1e584ca329e84d5e8d9149baa4eb28d2/users/585990cfe98f4a8187f020b3f4ee316f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:41.895200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e584ca329e84d5e8d9149baa4eb28d2, user_id=585990cfe98f4a8187f020b3f4ee316f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:41.899087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:41.913569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1316/2653] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:41 2022] PUT /identity/v3/projects/1e584ca329e84d5e8d9149baa4eb28d2/users/585990cfe98f4a8187f020b3f4ee316f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:41.917795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:41.928004 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d77a5-489c-4317-977b-354d8795edf1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:41.984951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d5b976-539b-4304-96bc-c2e3ef77ad2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:41.985075 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5b976-539b-4304-96bc-c2e3ef77ad2a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:41.985337 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5b976-539b-4304-96bc-c2e3ef77ad2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:41.985567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d5b976-539b-4304-96bc-c2e3ef77ad2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:41.986207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d5b976-539b-4304-96bc-c2e3ef77ad2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:41.990256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d5b976-539b-4304-96bc-c2e3ef77ad2a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:41.998322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1338/2654] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:41 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:42.002603 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:42.013456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcba007-9dcf-4508-9489-9baec906f0f2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:42.018260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:42.018921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:42.019196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:42.019468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] PATH_INFO: `/v3/projects/1e584ca329e84d5e8d9149baa4eb28d2/users/585990cfe98f4a8187f020b3f4ee316f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:42.021666 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e584ca329e84d5e8d9149baa4eb28d2, user_id=585990cfe98f4a8187f020b3f4ee316f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:42.024805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:42.042757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1317/2655] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:42 2022] PUT /identity/v3/projects/1e584ca329e84d5e8d9149baa4eb28d2/users/585990cfe98f4a8187f020b3f4ee316f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:42.672479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fe7c67-7860-4e7d-89fd-249c9d01aa55 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:42.672768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fe7c67-7860-4e7d-89fd-249c9d01aa55 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:42.673077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fe7c67-7860-4e7d-89fd-249c9d01aa55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:42.701752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-60fe7c67-7860-4e7d-89fd-249c9d01aa55 None None] MFA Rules not processed for user `585990cfe98f4a8187f020b3f4ee316f`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:42.702973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-60fe7c67-7860-4e7d-89fd-249c9d01aa55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:42.779700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1339/2656] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:42 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:43.986502 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:43.988237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:44.062315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fe7c67-7860-4e7d-89fd-249c9d01aa55 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:44.062813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cad122-caec-4319-8ad0-e7e1c589c735 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:44.068690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:44.070062 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:44.070375 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:44.070712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:44.072472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:44.084225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:44.085923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:44.096517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1318/2657] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:24:44.132848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fe7c67-7860-4e7d-89fd-249c9d01aa55 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:44.147138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:44.147138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:44.147138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:44.147138 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] PATH_INFO: `/v3/users/fe362ac84d4d46778ab386d1aa9b9a4a` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:44.147675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe362ac84d4d46778ab386d1aa9b9a4a)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:44.147675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:44.166946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:44.197012 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe362ac84d4d46778ab386d1aa9b9a4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.203706 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe362ac84d4d46778ab386d1aa9b9a4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.204376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe362ac84d4d46778ab386d1aa9b9a4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.205130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe362ac84d4d46778ab386d1aa9b9a4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.212601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe362ac84d4d46778ab386d1aa9b9a4a'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.232604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1340/2658] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:44 2022] DELETE /identity/v3/users/fe362ac84d4d46778ab386d1aa9b9a4a => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:24:44.401121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:44.412400 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6970687d-d2a7-483c-88d8-402d7e25fed5 service placement] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:44.456249 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:44.473243 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:44.473876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:44.474161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:44.474436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] PATH_INFO: `/v3/projects/c159165317bc41b38f7155d638072688` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:44.476154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c159165317bc41b38f7155d638072688)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:44.478086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:44.529152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c159165317bc41b38f7155d638072688', 'request_id': 'req-a43c7538-c764-4af7-902f-7d84fe746c15'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.534791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c159165317bc41b38f7155d638072688', 'request_id': 'req-a43c7538-c764-4af7-902f-7d84fe746c15'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.543192 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c159165317bc41b38f7155d638072688', 'request_id': 'req-a43c7538-c764-4af7-902f-7d84fe746c15'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.555176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] The token cache is being invalidate because project c159165317bc41b38f7155d638072688 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:44.555465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.556092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.557569 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1319/2659] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:44 2022] DELETE /identity/v3/projects/c159165317bc41b38f7155d638072688 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:44.561878 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:44.563297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:44.576149 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789ae65-d45f-4807-867b-87a4d8dc119c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:44.632304 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:44.632963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:44.633245 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:44.633524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] PATH_INFO: `/v3/users/585990cfe98f4a8187f020b3f4ee316f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:44.634677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] RBAC: Authorizing `identity:delete_user(user_id=585990cfe98f4a8187f020b3f4ee316f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:44.636592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:44.684813 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '585990cfe98f4a8187f020b3f4ee316f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.691756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '585990cfe98f4a8187f020b3f4ee316f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.692446 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '585990cfe98f4a8187f020b3f4ee316f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.693207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '585990cfe98f4a8187f020b3f4ee316f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.700477 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-13719301-8e6d-4b66-b03d-0a9bcf933ccd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '585990cfe98f4a8187f020b3f4ee316f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:44.717961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1341/2660] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:44 2022] DELETE /identity/v3/users/585990cfe98f4a8187f020b3f4ee316f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:44.891447 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:44.892870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:44.911562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43c7538-c764-4af7-902f-7d84fe746c15 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:44.968425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:44.969084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:44.969364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:44.969632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] PATH_INFO: `/v3/projects/1e584ca329e84d5e8d9149baa4eb28d2` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:44.970726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e584ca329e84d5e8d9149baa4eb28d2)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:44.972642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.017693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e584ca329e84d5e8d9149baa4eb28d2', 'request_id': 'req-3b7f1350-fa96-4371-9dfa-055d5808bb06'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:45.023509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e584ca329e84d5e8d9149baa4eb28d2', 'request_id': 'req-3b7f1350-fa96-4371-9dfa-055d5808bb06'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:45.030407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e584ca329e84d5e8d9149baa4eb28d2', 'request_id': 'req-3b7f1350-fa96-4371-9dfa-055d5808bb06'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:45.042255 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] The token cache is being invalidate because project 1e584ca329e84d5e8d9149baa4eb28d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:45.042545 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:45.043182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3b7f1350-fa96-4371-9dfa-055d5808bb06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:45.044649 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1320/2661] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:44 2022] DELETE /identity/v3/projects/1e584ca329e84d5e8d9149baa4eb28d2 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:45.049612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee572490-9cf6-4ff9-8cd5-6a32b28c75cd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.049903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee572490-9cf6-4ff9-8cd5-6a32b28c75cd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.050183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee572490-9cf6-4ff9-8cd5-6a32b28c75cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.070564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ee572490-9cf6-4ff9-8cd5-6a32b28c75cd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:45.071463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ee572490-9cf6-4ff9-8cd5-6a32b28c75cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:45.085009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:45.132222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1342/2662] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:45.139922 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3a88df-7fc3-4868-8a85-be13e4f67d36 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.140211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3a88df-7fc3-4868-8a85-be13e4f67d36 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.140488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3a88df-7fc3-4868-8a85-be13e4f67d36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.149401 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:45.163077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ce3a88df-7fc3-4868-8a85-be13e4f67d36 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:45.165326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ce3a88df-7fc3-4868-8a85-be13e4f67d36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:45.173450 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1321/2663] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:45.178109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee572490-9cf6-4ff9-8cd5-6a32b28c75cd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.189169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee572490-9cf6-4ff9-8cd5-6a32b28c75cd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.194689 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.195301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.195597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.195848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.196493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:24:45.196795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.198623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.210676 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1343/2664] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:24:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:24:45.214994 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3a88df-7fc3-4868-8a85-be13e4f67d36 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.224542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3a88df-7fc3-4868-8a85-be13e4f67d36 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.229916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.230504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.230802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.231071 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.231704 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.233609 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.260906 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1322/2665] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/projects => generated 363 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:24:45.265046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.275133 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f5c615-8dda-418a-86a9-a10f4d6e7751 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.330961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.331491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.331770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.332041 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.332670 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.334797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.355324 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1344/2666] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/users => generated 282 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:45.359527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.370783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c95317-386c-4111-9878-30160de5f586 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.375343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.375931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.376211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.376482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.377152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.378715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.389300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1323/2667] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:45.393105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.403802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010509fa-cdc0-4e02-a28e-983943f1d7d7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.408436 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421e747e-a34c-46a9-983b-899248857993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.409058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421e747e-a34c-46a9-983b-899248857993 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.409341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421e747e-a34c-46a9-983b-899248857993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.409617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421e747e-a34c-46a9-983b-899248857993 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.410252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421e747e-a34c-46a9-983b-899248857993 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.411836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421e747e-a34c-46a9-983b-899248857993 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.422982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1345/2668] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:45.427006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.436559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e18e54-0199-48ce-9552-6b2c80e311b7 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.441073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.441708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.441986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.442256 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] PATH_INFO: `/v3/projects/e2a7918517544fd389419259390fb7bf/users/7bfc2dd4b7ad4442b94237dd892fd3a7/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.456043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2a7918517544fd389419259390fb7bf, user_id=7bfc2dd4b7ad4442b94237dd892fd3a7, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.458682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.471728 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1324/2669] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:45 2022] PUT /identity/v3/projects/e2a7918517544fd389419259390fb7bf/users/7bfc2dd4b7ad4442b94237dd892fd3a7/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:45.483532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851726eb-1d38-4310-8803-660fcbd03385 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.483820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851726eb-1d38-4310-8803-660fcbd03385 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.484102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851726eb-1d38-4310-8803-660fcbd03385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.512353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-851726eb-1d38-4310-8803-660fcbd03385 None None] MFA Rules not processed for user `7bfc2dd4b7ad4442b94237dd892fd3a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:45.513598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-851726eb-1d38-4310-8803-660fcbd03385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:45.576838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1346/2670] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:45.581303 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.640983 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157ff02d-1265-4161-af73-2a9e95dc57f1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.645931 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.646559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.647338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.647834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.648315 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.650240 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.676715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1325/2671] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/projects => generated 365 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:45.680831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851726eb-1d38-4310-8803-660fcbd03385 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.692444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851726eb-1d38-4310-8803-660fcbd03385 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.701886 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:45.750443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.751050 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.751333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.751615 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.752253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.754195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.775929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1347/2672] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/users => generated 289 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:45.779975 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.792377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b5d92-d448-4f6b-bc98-0776dfee7c0b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.796996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.797590 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.797872 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.798142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.798774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.800341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.810894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1326/2673] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:45.815049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.826945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe332ec1-54e5-4b14-92d4-9fbfef84a17b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.832547 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.833173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.833461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.833741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] PATH_INFO: `/v3/projects/907c164e0038438e9ac23ada6759ce23/users/a45ecffffd6c4d6a8dbcfdebd9ce9eaa/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.849330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] RBAC: Authorizing `identity:create_grant(project_id=907c164e0038438e9ac23ada6759ce23, user_id=a45ecffffd6c4d6a8dbcfdebd9ce9eaa, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.852018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.864681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1348/2674] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:45 2022] PUT /identity/v3/projects/907c164e0038438e9ac23ada6759ce23/users/a45ecffffd6c4d6a8dbcfdebd9ce9eaa/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:45.868694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.878547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e28c8-1db6-4773-b2bf-a2f9a63209b9 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.939276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e13d10-3d1b-4458-8e92-0c73cc28659c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.939876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e13d10-3d1b-4458-8e92-0c73cc28659c admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.940156 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e13d10-3d1b-4458-8e92-0c73cc28659c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.940421 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e13d10-3d1b-4458-8e92-0c73cc28659c admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.941083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e13d10-3d1b-4458-8e92-0c73cc28659c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.942941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e13d10-3d1b-4458-8e92-0c73cc28659c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.953310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1327/2675] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:45 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:45.957331 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:45.968119 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c451dfd-9edd-4c09-8bc6-2c5706b99524 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:45.972746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:45.973428 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.973708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.973981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] PATH_INFO: `/v3/projects/907c164e0038438e9ac23ada6759ce23/users/a45ecffffd6c4d6a8dbcfdebd9ce9eaa/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:45.976009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] RBAC: Authorizing `identity:create_grant(project_id=907c164e0038438e9ac23ada6759ce23, user_id=a45ecffffd6c4d6a8dbcfdebd9ce9eaa, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:45.978682 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:45.988548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1349/2676] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:45 2022] PUT /identity/v3/projects/907c164e0038438e9ac23ada6759ce23/users/a45ecffffd6c4d6a8dbcfdebd9ce9eaa/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:45.998554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:45.998847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:45.999127 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:46.019561 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:46.028309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] MFA Rules not processed for user `a45ecffffd6c4d6a8dbcfdebd9ce9eaa`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:46.029830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:46.098322 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1328/2677] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:45 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:46.109607 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:46.111419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:46.170432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f93680d-b26c-4271-bfc2-9601a370e669 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:46.174069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:46.174668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:46.174947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:46.175221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:46.184997 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:46.186421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:46.196792 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1350/2678] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:46 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:46.281486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:46.282873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:46.295339 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af6c9f7-2684-41dd-8ba9-303cd8dab66d None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:46.298942 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bc3926-6442-4358-8fc1-a221b1dbf6ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:46.299539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc3926-6442-4358-8fc1-a221b1dbf6ef service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:46.299819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc3926-6442-4358-8fc1-a221b1dbf6ef service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:46.300094 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc3926-6442-4358-8fc1-a221b1dbf6ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:46.354768 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bc3926-6442-4358-8fc1-a221b1dbf6ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:46.356132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bc3926-6442-4358-8fc1-a221b1dbf6ef service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:46.364761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1329/2679] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:46 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:55.704241 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:55.766192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9b0f43-8b19-4a39-891a-771e42b54657 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:55.770983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:55.771718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:55.771998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:55.772273 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] PATH_INFO: `/v3/users/7bfc2dd4b7ad4442b94237dd892fd3a7` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:55.773541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bfc2dd4b7ad4442b94237dd892fd3a7)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:55.775430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:55.787707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:55.824131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bfc2dd4b7ad4442b94237dd892fd3a7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:55.830834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bfc2dd4b7ad4442b94237dd892fd3a7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:55.831555 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bfc2dd4b7ad4442b94237dd892fd3a7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:55.832158 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bfc2dd4b7ad4442b94237dd892fd3a7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:55.839605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bfc2dd4b7ad4442b94237dd892fd3a7'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:55.857701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1351/2680] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:55 2022] DELETE /identity/v3/users/7bfc2dd4b7ad4442b94237dd892fd3a7 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:55.862199 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:55.862682 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:55.862887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:55.882246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:55.883431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:55.942487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1330/2681] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:55.952494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:55.953938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:56.010921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67709a7f-28b3-4c9c-b3a9-f06f1703474d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:56.014626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:56.015229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:56.015510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:56.015895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:56.025678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:56.027060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:56.036346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1352/2682] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:56.391918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:56.393814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:56.406850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f90bff-4ad8-49cc-8f29-0f494d753aec None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:56.411633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:56.413155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:56.413534 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:56.413919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] PATH_INFO: `/v3/projects/e2a7918517544fd389419259390fb7bf` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:56.415181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2a7918517544fd389419259390fb7bf)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:56.417214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:56.453149 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:56.467253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2a7918517544fd389419259390fb7bf', 'request_id': 'req-130f7256-49d3-4876-b274-506a4994dedf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.473481 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2a7918517544fd389419259390fb7bf', 'request_id': 'req-130f7256-49d3-4876-b274-506a4994dedf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.481217 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2a7918517544fd389419259390fb7bf', 'request_id': 'req-130f7256-49d3-4876-b274-506a4994dedf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.492219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] The token cache is being invalidate because project e2a7918517544fd389419259390fb7bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:56.492511 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.493180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.494661 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1331/2683] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:56 2022] DELETE /identity/v3/projects/e2a7918517544fd389419259390fb7bf => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:56.498620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:56.508779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b8df5-fc2a-4994-92e5-6709953cc6ba service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:56.564814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:56.565479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:56.565762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:56.566038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] PATH_INFO: `/v3/users/a45ecffffd6c4d6a8dbcfdebd9ce9eaa` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:56.567153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a45ecffffd6c4d6a8dbcfdebd9ce9eaa)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:56.569190 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:56.603458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:56.616870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a45ecffffd6c4d6a8dbcfdebd9ce9eaa'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.623745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a45ecffffd6c4d6a8dbcfdebd9ce9eaa'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.624403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a45ecffffd6c4d6a8dbcfdebd9ce9eaa'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.625044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a45ecffffd6c4d6a8dbcfdebd9ce9eaa'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.632946 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-4c10e902-84bf-4d41-bb6c-59e38179182a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a45ecffffd6c4d6a8dbcfdebd9ce9eaa'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:56.649994 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1353/2684] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:24:56 2022] DELETE /identity/v3/users/a45ecffffd6c4d6a8dbcfdebd9ce9eaa => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:57.113803 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:57.116321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:57.130509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f7256-49d3-4876-b274-506a4994dedf admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:57.214822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:57.215504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.215876 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.216152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] PATH_INFO: `/v3/projects/907c164e0038438e9ac23ada6759ce23` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.217884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] RBAC: Authorizing `identity:delete_project(project_id=907c164e0038438e9ac23ada6759ce23)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:57.220030 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:57.287951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '907c164e0038438e9ac23ada6759ce23', 'request_id': 'req-55b219b7-5454-4ede-8bab-330f6281ebdf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:57.293967 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '907c164e0038438e9ac23ada6759ce23', 'request_id': 'req-55b219b7-5454-4ede-8bab-330f6281ebdf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:57.302090 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '907c164e0038438e9ac23ada6759ce23', 'request_id': 'req-55b219b7-5454-4ede-8bab-330f6281ebdf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:57.318083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] The token cache is being invalidate because project 907c164e0038438e9ac23ada6759ce23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:24:57.318437 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:57.319377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-55b219b7-5454-4ede-8bab-330f6281ebdf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:24:57.321061 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1332/2685] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:24:57 2022] DELETE /identity/v3/projects/907c164e0038438e9ac23ada6759ce23 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:57.328287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca21022c-e1b9-4c66-8729-8beda93828c0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.328789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca21022c-e1b9-4c66-8729-8beda93828c0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.328889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca21022c-e1b9-4c66-8729-8beda93828c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.352058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ca21022c-e1b9-4c66-8729-8beda93828c0 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:57.353503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ca21022c-e1b9-4c66-8729-8beda93828c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:57.420152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1354/2686] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:57.430426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858d8eb-5baf-4bf6-addd-778da5a8ee2e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.430715 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858d8eb-5baf-4bf6-addd-778da5a8ee2e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.431037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d858d8eb-5baf-4bf6-addd-778da5a8ee2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.452544 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-d858d8eb-5baf-4bf6-addd-778da5a8ee2e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:57.454951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d858d8eb-5baf-4bf6-addd-778da5a8ee2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:57.465562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1333/2687] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:57.469554 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca21022c-e1b9-4c66-8729-8beda93828c0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:57.481715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca21022c-e1b9-4c66-8729-8beda93828c0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:57.488294 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78399d47-3bad-4280-9358-068311030985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:57.490373 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78399d47-3bad-4280-9358-068311030985 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.491104 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78399d47-3bad-4280-9358-068311030985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.491391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78399d47-3bad-4280-9358-068311030985 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.492116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78399d47-3bad-4280-9358-068311030985 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:24:57.492465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78399d47-3bad-4280-9358-068311030985 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:57.494140 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78399d47-3bad-4280-9358-068311030985 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:57.516877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1355/2688] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:24:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:24:57.522242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d858d8eb-5baf-4bf6-addd-778da5a8ee2e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:57.533896 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d858d8eb-5baf-4bf6-addd-778da5a8ee2e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:57.540519 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:57.541259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.541621 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.541957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.542671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:57.544644 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:57.578694 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1334/2689] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:57 2022] POST /identity/v3/projects => generated 377 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:57.584975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78399d47-3bad-4280-9358-068311030985 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:57.595192 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78399d47-3bad-4280-9358-068311030985 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:57.671180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:57.671802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.672083 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.672356 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.673035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:57.674962 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:57.704505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1356/2690] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:57 2022] POST /identity/v3/users => generated 289 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:57.708584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:57.725245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17642c46-d1b4-4dfc-9e27-ab678fb1725c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:57.730246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:57.730905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.731190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.731462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.732109 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:57.733740 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:57.746788 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1335/2691] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:57.751504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:57.765336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075b2dd4-4548-4e8c-9601-9a09f4297a52 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:57.770645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd43559-ecfa-4d95-b7e2-4d9ab8399202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:57.771758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd43559-ecfa-4d95-b7e2-4d9ab8399202 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.772188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd43559-ecfa-4d95-b7e2-4d9ab8399202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.772465 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd43559-ecfa-4d95-b7e2-4d9ab8399202 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.773131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd43559-ecfa-4d95-b7e2-4d9ab8399202 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:57.774794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd43559-ecfa-4d95-b7e2-4d9ab8399202 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:57.786510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1357/2692] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:57 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:57.791453 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:57.804532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d94f286-70df-47d1-ab7a-4e2cc0cac946 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:57.809261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:57.809925 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.810208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.810486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] PATH_INFO: `/v3/projects/08ed0b8ad08940deb59e35b8f4a735b1/users/2450d69f54a74415aa81ac7b37068259/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.814430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:57.833194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08ed0b8ad08940deb59e35b8f4a735b1, user_id=2450d69f54a74415aa81ac7b37068259, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:57.836460 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:57.849499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1336/2693] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:57 2022] PUT /identity/v3/projects/08ed0b8ad08940deb59e35b8f4a735b1/users/2450d69f54a74415aa81ac7b37068259/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:24:57.860667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003e3e3-b814-42ae-8b04-aece81521190 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:57.860975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003e3e3-b814-42ae-8b04-aece81521190 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:57.861254 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003e3e3-b814-42ae-8b04-aece81521190 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:57.889617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-3003e3e3-b814-42ae-8b04-aece81521190 None None] MFA Rules not processed for user `2450d69f54a74415aa81ac7b37068259`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:57.890782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3003e3e3-b814-42ae-8b04-aece81521190 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:57.894775 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:57.956072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1358/2694] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:57 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:57.961433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.025822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b861e53b-5a67-457b-b464-aa14e8a36625 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.030552 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.031159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.031440 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.031764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.032397 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.034325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.062372 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1337/2695] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:57 2022] POST /identity/v3/projects => generated 377 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:58.066486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3003e3e3-b814-42ae-8b04-aece81521190 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.077505 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3003e3e3-b814-42ae-8b04-aece81521190 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.135232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.135842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.136121 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.136393 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.137057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.138963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.161575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1359/2696] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:58 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:58.166259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.176554 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad153de-629b-454f-b857-237ed786f6b0 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.181338 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.181937 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.182218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.182492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.183128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.184712 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.195606 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1338/2697] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:58.199620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.211155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37d2648-c28f-4bd6-a801-b507e9e5d520 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.215842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bf8c32-c61f-422b-bb9a-cbedd3add89b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.216444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf8c32-c61f-422b-bb9a-cbedd3add89b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.216728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf8c32-c61f-422b-bb9a-cbedd3add89b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.217086 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf8c32-c61f-422b-bb9a-cbedd3add89b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.217734 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bf8c32-c61f-422b-bb9a-cbedd3add89b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.219354 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bf8c32-c61f-422b-bb9a-cbedd3add89b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.231357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1360/2698] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:58.235693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.245792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8e2d90-790d-4611-8f5d-a51a3b64f2e1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.250508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.251145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.251425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.251700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] PATH_INFO: `/v3/projects/6833b7b3139e44df8a64ebe8f0630c6f/users/8d5f150c130e41afa9656023e5bc8829/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.265946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] RBAC: Authorizing `identity:create_grant(project_id=6833b7b3139e44df8a64ebe8f0630c6f, user_id=8d5f150c130e41afa9656023e5bc8829, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.268587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.282651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1339/2699] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:58 2022] PUT /identity/v3/projects/6833b7b3139e44df8a64ebe8f0630c6f/users/8d5f150c130e41afa9656023e5bc8829/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:58.293633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6851e32-e8d1-4271-b5fb-b6cfd05318b1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.293925 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6851e32-e8d1-4271-b5fb-b6cfd05318b1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.294203 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6851e32-e8d1-4271-b5fb-b6cfd05318b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.321890 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-b6851e32-e8d1-4271-b5fb-b6cfd05318b1 None None] MFA Rules not processed for user `8d5f150c130e41afa9656023e5bc8829`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:58.323091 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b6851e32-e8d1-4271-b5fb-b6cfd05318b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:58.399376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1361/2700] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:24:58.403847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.466273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03c2c1d-8f16-4656-8c6b-5bae2fe42fbc admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.471228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.471824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.472129 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.472402 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.473104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.475559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.501726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1340/2701] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:24:58 2022] POST /identity/v3/projects => generated 377 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:24:58.506058 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6851e32-e8d1-4271-b5fb-b6cfd05318b1 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.517106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6851e32-e8d1-4271-b5fb-b6cfd05318b1 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.527131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:58.574887 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.575511 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.575769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.576035 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.576653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.578597 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.598909 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1362/2702] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:24:58 2022] POST /identity/v3/users => generated 295 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:24:58.603078 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.613295 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048e9c31-0572-4b78-9ace-e09a145fe105 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.618224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.618849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.619134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.619414 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.620057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.621681 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.634688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1341/2703] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:58.639125 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.650398 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52e6aa-d24c-4894-baae-a8c4597d0a77 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.654996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.655647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.656225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.656333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] PATH_INFO: `/v3/projects/265ea8d7001f4f70953207bf14e37b47/users/6dcf27e13d1d46c58f848eb148535093/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.672824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=265ea8d7001f4f70953207bf14e37b47, user_id=6dcf27e13d1d46c58f848eb148535093, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.675456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.688616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1363/2704] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:58 2022] PUT /identity/v3/projects/265ea8d7001f4f70953207bf14e37b47/users/6dcf27e13d1d46c58f848eb148535093/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:24:58.693744 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.703992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef5f305-ea20-4a59-b232-da0b3a70ce1f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.709452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:24:58.769503 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6920c28-fc25-4cf7-a7ec-562253454cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.770370 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6920c28-fc25-4cf7-a7ec-562253454cb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.770653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6920c28-fc25-4cf7-a7ec-562253454cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.770924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6920c28-fc25-4cf7-a7ec-562253454cb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.772196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6920c28-fc25-4cf7-a7ec-562253454cb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.773816 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6920c28-fc25-4cf7-a7ec-562253454cb8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.786991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1342/2705] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:24:58 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:24:58.788594 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.799130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a9d78-b224-4156-887e-0abda0f45e89 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:58.803967 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:58.804641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.804918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.805222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] PATH_INFO: `/v3/projects/265ea8d7001f4f70953207bf14e37b47/users/6dcf27e13d1d46c58f848eb148535093/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.807108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] RBAC: Authorizing `identity:create_grant(project_id=265ea8d7001f4f70953207bf14e37b47, user_id=6dcf27e13d1d46c58f848eb148535093, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:58.810038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:58.821874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1364/2706] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:24:58 2022] PUT /identity/v3/projects/265ea8d7001f4f70953207bf14e37b47/users/6dcf27e13d1d46c58f848eb148535093/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:24:58.832015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:58.832307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:58.832584 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:58.860433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] MFA Rules not processed for user `6dcf27e13d1d46c58f848eb148535093`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:24:58.861693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:58.930133 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1343/2707] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:24:58 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:24:58.942200 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:58.944898 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:59.004649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9981e-3e7d-4e60-a212-5f7add45b970 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:59.009588 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:59.010208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:59.010490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:59.010879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:59.021902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:59.023334 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:59.032605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1365/2708] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:58 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:24:59.075230 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:24:59.076640 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:24:59.088691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1effc-bc6f-4ea4-a35e-c0390f046c30 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:24:59.092332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dac4897-6ae6-4a9e-924c-257af011be9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:24:59.092933 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dac4897-6ae6-4a9e-924c-257af011be9a service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:24:59.093239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dac4897-6ae6-4a9e-924c-257af011be9a service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:24:59.093513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dac4897-6ae6-4a9e-924c-257af011be9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:24:59.150759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dac4897-6ae6-4a9e-924c-257af011be9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:24:59.152245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dac4897-6ae6-4a9e-924c-257af011be9a service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:24:59.161175 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1344/2709] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:24:59 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:02.065763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:02.126801 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ead0d52-af31-4dfa-b312-3a7436ecdf18 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:02.131833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:02.132451 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:02.132727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:02.133028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] PATH_INFO: `/v3/users/2450d69f54a74415aa81ac7b37068259` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:02.134146 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2450d69f54a74415aa81ac7b37068259)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:02.136049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:02.146981 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:02.183484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2450d69f54a74415aa81ac7b37068259'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.190308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2450d69f54a74415aa81ac7b37068259'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.191002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2450d69f54a74415aa81ac7b37068259'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.191612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2450d69f54a74415aa81ac7b37068259'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.199486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2450d69f54a74415aa81ac7b37068259'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.220060 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1366/2710] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:02 2022] DELETE /identity/v3/users/2450d69f54a74415aa81ac7b37068259 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:02.224535 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:02.225112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:02.225112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:02.246106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:02.247650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:02.275607 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:02.327589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1345/2711] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:02.338197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:02.339956 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:02.410675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28383d8a-4bba-4848-8993-d79e85ba4a11 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:02.415018 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:02.415768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:02.416049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:02.416319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:02.429802 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:02.431259 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:02.455771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1367/2712] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:02.763350 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:02.764949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:02.784613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125a9874-3c7e-4917-9b4b-3395ea8b7b8e None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:02.797118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:02.797805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:02.798100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:02.798383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] PATH_INFO: `/v3/projects/08ed0b8ad08940deb59e35b8f4a735b1` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:02.799564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=08ed0b8ad08940deb59e35b8f4a735b1)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:02.801504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:02.850904 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08ed0b8ad08940deb59e35b8f4a735b1', 'request_id': 'req-3511365b-d579-4234-bb67-30408ce6ef9f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.856615 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08ed0b8ad08940deb59e35b8f4a735b1', 'request_id': 'req-3511365b-d579-4234-bb67-30408ce6ef9f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.863895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08ed0b8ad08940deb59e35b8f4a735b1', 'request_id': 'req-3511365b-d579-4234-bb67-30408ce6ef9f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.875907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] The token cache is being invalidate because project 08ed0b8ad08940deb59e35b8f4a735b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:02.876278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.876969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:02.878491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1346/2713] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:02 2022] DELETE /identity/v3/projects/08ed0b8ad08940deb59e35b8f4a735b1 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:02.882917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:02.892721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6521031c-0343-45cf-a92f-77201089bf12 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:02.952648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:02.953314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:02.953598 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:02.953874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] PATH_INFO: `/v3/users/8d5f150c130e41afa9656023e5bc8829` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:02.954993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d5f150c130e41afa9656023e5bc8829)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:02.956912 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:02.990883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:03.004243 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d5f150c130e41afa9656023e5bc8829'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.011895 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d5f150c130e41afa9656023e5bc8829'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.012558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d5f150c130e41afa9656023e5bc8829'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.013193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d5f150c130e41afa9656023e5bc8829'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.020907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d5f150c130e41afa9656023e5bc8829'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.038238 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1368/2714] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:02 2022] DELETE /identity/v3/users/8d5f150c130e41afa9656023e5bc8829 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:03.316269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:03.317683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:03.331291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3511365b-d579-4234-bb67-30408ce6ef9f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:03.387680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:03.388302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:03.388583 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:03.388854 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] PATH_INFO: `/v3/projects/6833b7b3139e44df8a64ebe8f0630c6f` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:03.389997 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6833b7b3139e44df8a64ebe8f0630c6f)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:03.391885 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:03.441046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6833b7b3139e44df8a64ebe8f0630c6f', 'request_id': 'req-a0ac7656-2a75-44ea-afc2-84405455b245'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.445113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:03.447553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6833b7b3139e44df8a64ebe8f0630c6f', 'request_id': 'req-a0ac7656-2a75-44ea-afc2-84405455b245'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.456265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6833b7b3139e44df8a64ebe8f0630c6f', 'request_id': 'req-a0ac7656-2a75-44ea-afc2-84405455b245'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.469037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] The token cache is being invalidate because project 6833b7b3139e44df8a64ebe8f0630c6f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:03.469342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.469974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.471464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1347/2715] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:03 2022] DELETE /identity/v3/projects/6833b7b3139e44df8a64ebe8f0630c6f => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:03.476379 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:03.478414 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:03.491105 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a178dbe-74ac-413d-8612-9ed4ad127fc6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:03.554157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:03.554870 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:03.555179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:03.555531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] PATH_INFO: `/v3/users/6dcf27e13d1d46c58f848eb148535093` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:03.556646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6dcf27e13d1d46c58f848eb148535093)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:03.558641 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:03.604885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6dcf27e13d1d46c58f848eb148535093'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.611612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6dcf27e13d1d46c58f848eb148535093'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.612267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6dcf27e13d1d46c58f848eb148535093'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.612868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6dcf27e13d1d46c58f848eb148535093'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.620206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-1b2b3bed-4c04-43ac-be89-3c4040a82714 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6dcf27e13d1d46c58f848eb148535093'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:03.636977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1369/2716] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:03 2022] DELETE /identity/v3/users/6dcf27e13d1d46c58f848eb148535093 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:03.905262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:03.906706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:03.919496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ac7656-2a75-44ea-afc2-84405455b245 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:03.983936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:03.984586 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:03.984870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:03.985182 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] PATH_INFO: `/v3/projects/265ea8d7001f4f70953207bf14e37b47` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:03.986882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] RBAC: Authorizing `identity:delete_project(project_id=265ea8d7001f4f70953207bf14e37b47)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:03.988824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.039410 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '265ea8d7001f4f70953207bf14e37b47', 'request_id': 'req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:04.045717 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '265ea8d7001f4f70953207bf14e37b47', 'request_id': 'req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:04.053496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '265ea8d7001f4f70953207bf14e37b47', 'request_id': 'req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:04.064834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] The token cache is being invalidate because project 265ea8d7001f4f70953207bf14e37b47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:04.065773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:04.066430 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ff600ca6-99b7-4b12-84a7-5dc6be8b4650 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:04.070073 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1348/2717] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:03 2022] DELETE /identity/v3/projects/265ea8d7001f4f70953207bf14e37b47 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:04.074868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf9d05c-dd07-4565-8d72-13d4a41d07ee None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.075089 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf9d05c-dd07-4565-8d72-13d4a41d07ee None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.075359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf9d05c-dd07-4565-8d72-13d4a41d07ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.097623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-adf9d05c-dd07-4565-8d72-13d4a41d07ee None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:04.099080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-adf9d05c-dd07-4565-8d72-13d4a41d07ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:04.127619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:04.164462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1370/2718] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:04.172834 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987a392-f159-4043-a5f6-65e1433b191f None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.173155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987a392-f159-4043-a5f6-65e1433b191f None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.173433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987a392-f159-4043-a5f6-65e1433b191f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.194685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b987a392-f159-4043-a5f6-65e1433b191f None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:04.195888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b987a392-f159-4043-a5f6-65e1433b191f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:04.203760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1349/2719] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:25:04.207945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf9d05c-dd07-4565-8d72-13d4a41d07ee None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.217934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf9d05c-dd07-4565-8d72-13d4a41d07ee None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.223198 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.223933 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.224340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.224873 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.225852 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:25:04.226292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.228390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.240751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1371/2720] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:25:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:25:04.245273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b987a392-f159-4043-a5f6-65e1433b191f None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.254539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b987a392-f159-4043-a5f6-65e1433b191f None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.259748 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.260340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.260635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.260905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.261567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.263432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.289444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1350/2721] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:25:04 2022] POST /identity/v3/projects => generated 387 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:04.293585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.304562 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85f4fb0-1f7f-4a28-9604-ad21684e6da8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.363941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.364472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.365298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.365576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.366268 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.368236 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.389278 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1372/2722] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:25:04 2022] POST /identity/v3/users => generated 294 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:04.393426 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.403762 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c479195-0b3c-4dc7-9713-b971e317eaea admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.408321 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.408913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.409216 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.409486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.410118 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.411696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.421279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1351/2723] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:04 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:25:04.425014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.435383 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beaa2481-614a-4413-99b0-f35f0e850b06 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.439960 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cf132c-9fe9-43ea-b8b6-a8de77684c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.440549 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cf132c-9fe9-43ea-b8b6-a8de77684c44 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.440834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cf132c-9fe9-43ea-b8b6-a8de77684c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.441136 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cf132c-9fe9-43ea-b8b6-a8de77684c44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.441774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cf132c-9fe9-43ea-b8b6-a8de77684c44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.444256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cf132c-9fe9-43ea-b8b6-a8de77684c44 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.454619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1373/2724] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:25:04.458730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.468671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458436fa-7bb4-4149-917a-0faa677a3b82 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.473343 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.474020 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.474546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.474546 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] PATH_INFO: `/v3/projects/617c796c4d7d4210867335976bf5b4d3/users/0998e4700aee4542ba4271ccffb00b36/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.488758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=617c796c4d7d4210867335976bf5b4d3, user_id=0998e4700aee4542ba4271ccffb00b36, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.492219 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.504553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1352/2725] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:25:04 2022] PUT /identity/v3/projects/617c796c4d7d4210867335976bf5b4d3/users/0998e4700aee4542ba4271ccffb00b36/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:25:04.516079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69d691-30a6-472e-a5af-18d562a79add None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.516548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69d691-30a6-472e-a5af-18d562a79add None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.516669 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69d691-30a6-472e-a5af-18d562a79add None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.545394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-2f69d691-30a6-472e-a5af-18d562a79add None None] MFA Rules not processed for user `0998e4700aee4542ba4271ccffb00b36`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:04.546653 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2f69d691-30a6-472e-a5af-18d562a79add None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:04.614323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1374/2726] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:04.619121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.629184 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:04.692104 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a434157b-16ae-4d3d-a949-b79095702d4b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.699431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.700188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.701211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.701557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.702332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.704353 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.733747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1353/2727] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:25:04 2022] POST /identity/v3/projects => generated 387 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:04.737767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f69d691-30a6-472e-a5af-18d562a79add None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.749480 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f69d691-30a6-472e-a5af-18d562a79add None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.773309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:04.811326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.811935 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.812220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.812499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.813173 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.815099 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.837155 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1375/2728] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:25:04 2022] POST /identity/v3/users => generated 300 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:04.841142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.851275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea370f0-a59e-4ebd-b4bd-0a433507d218 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.855894 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.856486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.856766 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.857107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.857763 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.859351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.869806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1354/2729] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:25:04.873678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.883987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a39c3-0f87-4d2d-969e-f87d001b09c5 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.888541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.889213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.889521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.889794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] PATH_INFO: `/v3/projects/00f64d78fb344e05a076d86b20e5a76e/users/146d252d99c2448ba4648e7d494dac85/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.904897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00f64d78fb344e05a076d86b20e5a76e, user_id=146d252d99c2448ba4648e7d494dac85, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.907558 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:04.920415 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1376/2730] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:25:04 2022] PUT /identity/v3/projects/00f64d78fb344e05a076d86b20e5a76e/users/146d252d99c2448ba4648e7d494dac85/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:25:04.924687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:04.934858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee520e8-0cd6-492f-84d7-29730bb66b07 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:04.993364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217b6aac-05da-4b07-9ed1-ac389d2471c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:04.993996 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217b6aac-05da-4b07-9ed1-ac389d2471c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:04.994247 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217b6aac-05da-4b07-9ed1-ac389d2471c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:04.994515 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217b6aac-05da-4b07-9ed1-ac389d2471c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:04.995155 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217b6aac-05da-4b07-9ed1-ac389d2471c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:04.996733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217b6aac-05da-4b07-9ed1-ac389d2471c0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:05.009283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1355/2731] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:04 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:25:05.013537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:05.024224 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd25fb0-fdbe-4787-9daf-f2bb566df6d9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:05.028806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:05.029495 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:05.029779 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:05.030044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] PATH_INFO: `/v3/projects/00f64d78fb344e05a076d86b20e5a76e/users/146d252d99c2448ba4648e7d494dac85/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:05.032003 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] RBAC: Authorizing `identity:create_grant(project_id=00f64d78fb344e05a076d86b20e5a76e, user_id=146d252d99c2448ba4648e7d494dac85, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:05.034665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:05.045527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1377/2732] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:25:05 2022] PUT /identity/v3/projects/00f64d78fb344e05a076d86b20e5a76e/users/146d252d99c2448ba4648e7d494dac85/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:25:05.057220 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242c21ec-a117-46e4-82d8-1b2114a424db None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:05.057505 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242c21ec-a117-46e4-82d8-1b2114a424db None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:05.057778 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242c21ec-a117-46e4-82d8-1b2114a424db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:05.086580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-242c21ec-a117-46e4-82d8-1b2114a424db None None] MFA Rules not processed for user `146d252d99c2448ba4648e7d494dac85`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:05.087811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-242c21ec-a117-46e4-82d8-1b2114a424db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:05.166650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1356/2733] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:05 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:05.178618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:05.180442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:05.249712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd8d7c-375c-4ed5-bffb-24ffa86ec38b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:05.253879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f185fb-8265-45c9-8979-6477c7772fcc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:05.254528 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f185fb-8265-45c9-8979-6477c7772fcc service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:05.254811 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f185fb-8265-45c9-8979-6477c7772fcc service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:05.255232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f185fb-8265-45c9-8979-6477c7772fcc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:05.265735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f185fb-8265-45c9-8979-6477c7772fcc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:05.267340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f185fb-8265-45c9-8979-6477c7772fcc service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:05.280234 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1378/2734] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:05 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:06.002236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242c21ec-a117-46e4-82d8-1b2114a424db None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:06.008691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:06.182597 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242c21ec-a117-46e4-82d8-1b2114a424db None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:06.199340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:06.200528 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:06.201095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:06.202729 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] PATH_INFO: `/v3/users/0998e4700aee4542ba4271ccffb00b36` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:06.206476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0998e4700aee4542ba4271ccffb00b36)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:06.217147 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:06.346647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0998e4700aee4542ba4271ccffb00b36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:06.347653 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0998e4700aee4542ba4271ccffb00b36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:06.358239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0998e4700aee4542ba4271ccffb00b36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:06.359283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0998e4700aee4542ba4271ccffb00b36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:06.372392 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0998e4700aee4542ba4271ccffb00b36'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:06.426593 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1357/2735] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:06 2022] DELETE /identity/v3/users/0998e4700aee4542ba4271ccffb00b36 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:06.428908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91db57-8b9c-4bee-8c5c-e2f871915acd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:06.429424 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91db57-8b9c-4bee-8c5c-e2f871915acd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:06.429525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91db57-8b9c-4bee-8c5c-e2f871915acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:06.456229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9a91db57-8b9c-4bee-8c5c-e2f871915acd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:06.460188 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9a91db57-8b9c-4bee-8c5c-e2f871915acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:06.552749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1379/2736] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:06.562756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:06.564836 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:06.641674 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a91db57-8b9c-4bee-8c5c-e2f871915acd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:06.651616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529a781a-c5cb-42ae-951d-d8f1dfd980b7 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:06.655589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:06.656973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:06.657431 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:06.657638 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a91db57-8b9c-4bee-8c5c-e2f871915acd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:06.658015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:06.662816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:06.663491 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:06.663857 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:06.664231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:06.666360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:06.669543 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:06.670974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:06.683103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1358/2737] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:06.705325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:06.794131 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:06.795831 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:06.810476 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1380/2738] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:06 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:07.035633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:07.038464 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:07.053552 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b6c98-6d2d-4ba9-8cb9-d22c5044586f service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:07.058418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:07.059072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:07.059355 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:07.059668 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] PATH_INFO: `/v3/projects/617c796c4d7d4210867335976bf5b4d3` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:07.060806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] RBAC: Authorizing `identity:delete_project(project_id=617c796c4d7d4210867335976bf5b4d3)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:07.063813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:07.095434 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:07.134287 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '617c796c4d7d4210867335976bf5b4d3', 'request_id': 'req-f147b4ee-70f4-427e-b2fc-f7b0180c381e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.149024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '617c796c4d7d4210867335976bf5b4d3', 'request_id': 'req-f147b4ee-70f4-427e-b2fc-f7b0180c381e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.158527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '617c796c4d7d4210867335976bf5b4d3', 'request_id': 'req-f147b4ee-70f4-427e-b2fc-f7b0180c381e'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.173201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] The token cache is being invalidate because project 617c796c4d7d4210867335976bf5b4d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:07.173499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.174200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.176874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1359/2739] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:07 2022] DELETE /identity/v3/projects/617c796c4d7d4210867335976bf5b4d3 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:07.182209 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:07.200154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd8d22d-441e-40ed-a628-71e80adc3db2 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:07.271625 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:07.272283 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:07.272567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:07.272848 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] PATH_INFO: `/v3/users/146d252d99c2448ba4648e7d494dac85` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:07.274195 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] RBAC: Authorizing `identity:delete_user(user_id=146d252d99c2448ba4648e7d494dac85)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:07.276728 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:07.337782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '146d252d99c2448ba4648e7d494dac85'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.345913 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '146d252d99c2448ba4648e7d494dac85'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.346624 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '146d252d99c2448ba4648e7d494dac85'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.347237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '146d252d99c2448ba4648e7d494dac85'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.359454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-47f6a263-68a6-4ff9-bf88-108c56788442 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '146d252d99c2448ba4648e7d494dac85'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.380076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1381/2740] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:07 2022] DELETE /identity/v3/users/146d252d99c2448ba4648e7d494dac85 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:07.735290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:07.737474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:07.751119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f147b4ee-70f4-427e-b2fc-f7b0180c381e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:07.815367 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:07.816302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:07.816743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:07.817208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] PATH_INFO: `/v3/projects/00f64d78fb344e05a076d86b20e5a76e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:07.819969 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] RBAC: Authorizing `identity:delete_project(project_id=00f64d78fb344e05a076d86b20e5a76e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:07.822512 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:07.889627 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00f64d78fb344e05a076d86b20e5a76e', 'request_id': 'req-28adb995-9d94-4f97-8648-b1dfcebbcabf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.897012 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00f64d78fb344e05a076d86b20e5a76e', 'request_id': 'req-28adb995-9d94-4f97-8648-b1dfcebbcabf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.906250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00f64d78fb344e05a076d86b20e5a76e', 'request_id': 'req-28adb995-9d94-4f97-8648-b1dfcebbcabf'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.923829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] The token cache is being invalidate because project 00f64d78fb344e05a076d86b20e5a76e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:07.924276 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.925093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-28adb995-9d94-4f97-8648-b1dfcebbcabf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:07.931116 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1360/2741] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:07 2022] DELETE /identity/v3/projects/00f64d78fb344e05a076d86b20e5a76e => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:07.933044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f292a2-e4f3-453f-aa41-9e92acc50634 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:07.933525 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f292a2-e4f3-453f-aa41-9e92acc50634 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:07.933786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f292a2-e4f3-453f-aa41-9e92acc50634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:07.958034 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-64f292a2-e4f3-453f-aa41-9e92acc50634 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:07.959537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-64f292a2-e4f3-453f-aa41-9e92acc50634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:07.996571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:08.036982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1382/2742] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:08.045757 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdad057-bfb6-42cc-a7db-30c975ce02ac None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.046651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdad057-bfb6-42cc-a7db-30c975ce02ac None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.046932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdad057-bfb6-42cc-a7db-30c975ce02ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.069112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-4fdad057-bfb6-42cc-a7db-30c975ce02ac None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:08.070360 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-4fdad057-bfb6-42cc-a7db-30c975ce02ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:08.078579 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1361/2743] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:08.083782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f292a2-e4f3-453f-aa41-9e92acc50634 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.098504 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f292a2-e4f3-453f-aa41-9e92acc50634 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.104287 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.105286 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.105671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.106042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.106970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:25:08.107372 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.109124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.127127 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1383/2744] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:25:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:25:08.128683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdad057-bfb6-42cc-a7db-30c975ce02ac None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.138805 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdad057-bfb6-42cc-a7db-30c975ce02ac None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.144036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.144660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.145238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.145537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.146707 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.148838 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.182106 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1362/2745] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:25:08 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:08.187371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.200229 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb45e98-7267-4bfa-bb04-004352508fe0 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.277945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.279213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.279488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.279823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.280478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.283009 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.309552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1384/2746] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:25:08 2022] POST /identity/v3/users => generated 293 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:08.310759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.321842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7ed4c-af07-486a-85a9-6577ffa9fea8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.326441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.327128 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.327490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.327831 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.329190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.331019 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.337413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:08.345698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1363/2747] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:25:08.347721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.367819 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1e7a3-7224-41b8-83f9-413755afe14d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.372572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50645066-776a-4c1b-bb72-a1b22df2af0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.373280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50645066-776a-4c1b-bb72-a1b22df2af0a admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.373663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50645066-776a-4c1b-bb72-a1b22df2af0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.374298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50645066-776a-4c1b-bb72-a1b22df2af0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.374786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50645066-776a-4c1b-bb72-a1b22df2af0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.376475 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50645066-776a-4c1b-bb72-a1b22df2af0a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.388978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1385/2748] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:08 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:25:08.392888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.405107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b19b8a-3b64-4523-97bc-b0b55016b74d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.410845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.412293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.412630 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.412915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] PATH_INFO: `/v3/projects/309d1c987c8c49949f8a59b98223935e/users/604be04922b64e5d97b7ce4e0f42a57e/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.430651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=309d1c987c8c49949f8a59b98223935e, user_id=604be04922b64e5d97b7ce4e0f42a57e, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.433427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.465462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1364/2749] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:25:08 2022] PUT /identity/v3/projects/309d1c987c8c49949f8a59b98223935e/users/604be04922b64e5d97b7ce4e0f42a57e/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:08.466419 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0849da4-6435-4e8b-8186-3c42e734c5ff None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.466717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0849da4-6435-4e8b-8186-3c42e734c5ff None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.466984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0849da4-6435-4e8b-8186-3c42e734c5ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.496037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f0849da4-6435-4e8b-8186-3c42e734c5ff None None] MFA Rules not processed for user `604be04922b64e5d97b7ce4e0f42a57e`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:08.497313 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f0849da4-6435-4e8b-8186-3c42e734c5ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:08.579897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1386/2750] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:25:08.592559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.594383 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:08.664119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacf250e-a353-4e46-ac95-068619179ba4 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.668254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e43cca-6b4b-4b93-a2b1-d88cd70116e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.669011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e43cca-6b4b-4b93-a2b1-d88cd70116e1 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.669300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e43cca-6b4b-4b93-a2b1-d88cd70116e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.669575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e43cca-6b4b-4b93-a2b1-d88cd70116e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.679196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e43cca-6b4b-4b93-a2b1-d88cd70116e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.680637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e43cca-6b4b-4b93-a2b1-d88cd70116e1 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.690435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1365/2751] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:08.714521 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0849da4-6435-4e8b-8186-3c42e734c5ff None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.755232 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:08.791490 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0849da4-6435-4e8b-8186-3c42e734c5ff None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:08.796425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:08.797102 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.797380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.797650 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] PATH_INFO: `/v3/users/604be04922b64e5d97b7ce4e0f42a57e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.798774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=604be04922b64e5d97b7ce4e0f42a57e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:08.800721 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:08.855976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '604be04922b64e5d97b7ce4e0f42a57e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:08.867094 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '604be04922b64e5d97b7ce4e0f42a57e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:08.869853 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '604be04922b64e5d97b7ce4e0f42a57e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:08.871111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '604be04922b64e5d97b7ce4e0f42a57e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:08.879756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '604be04922b64e5d97b7ce4e0f42a57e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:08.907958 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1387/2752] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:08 2022] DELETE /identity/v3/users/604be04922b64e5d97b7ce4e0f42a57e => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:08.913475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:08.914041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:08.914533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:08.916617 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:08.918141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:08.938861 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:08.940152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:09.011489 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb1c03c-ab9d-4676-80e6-4e04b836dd12 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:09.013254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1366/2753] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:09.018220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:09.018770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:09.019046 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:09.019316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:09.022152 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:09.024736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:09.026226 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:09.035944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:09.037758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:09.049979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1388/2754] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:09.096131 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348e7027-149a-45dd-b028-3ac1ccee8478 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:09.100087 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3833dd-d76e-4bd6-b01e-07316612d761 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:09.100833 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3833dd-d76e-4bd6-b01e-07316612d761 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:09.100995 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3833dd-d76e-4bd6-b01e-07316612d761 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:09.101275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3833dd-d76e-4bd6-b01e-07316612d761 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:09.113850 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3833dd-d76e-4bd6-b01e-07316612d761 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:09.115619 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3833dd-d76e-4bd6-b01e-07316612d761 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:09.125993 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1367/2755] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:25:09.734701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:09.736377 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:09.751129 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a40b17-c368-485f-ba28-65470331b00b service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:09.756474 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:09.757348 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:09.757783 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:09.757897 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] PATH_INFO: `/v3/projects/309d1c987c8c49949f8a59b98223935e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:09.759031 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=309d1c987c8c49949f8a59b98223935e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:09.761124 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:09.828580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:09.854118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '309d1c987c8c49949f8a59b98223935e', 'request_id': 'req-bef33582-d1e8-4313-a283-c8b09f1a13b9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:09.862581 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '309d1c987c8c49949f8a59b98223935e', 'request_id': 'req-bef33582-d1e8-4313-a283-c8b09f1a13b9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:09.871542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '309d1c987c8c49949f8a59b98223935e', 'request_id': 'req-bef33582-d1e8-4313-a283-c8b09f1a13b9'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:09.890751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] The token cache is being invalidate because project 309d1c987c8c49949f8a59b98223935e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:09.891049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:09.891797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-bef33582-d1e8-4313-a283-c8b09f1a13b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:09.893472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1389/2756] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:09 2022] DELETE /identity/v3/projects/309d1c987c8c49949f8a59b98223935e => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:09.899629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e242c0-8a9e-460d-9f12-5b6f90327fae None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:09.899920 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e242c0-8a9e-460d-9f12-5b6f90327fae None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:09.900197 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e242c0-8a9e-460d-9f12-5b6f90327fae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:09.924622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b4e242c0-8a9e-460d-9f12-5b6f90327fae None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:09.925815 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b4e242c0-8a9e-460d-9f12-5b6f90327fae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:09.953575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:10.017562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1368/2757] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:10.088671 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa02e5-6713-4352-bbb8-82e3550683ba None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.089276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa02e5-6713-4352-bbb8-82e3550683ba None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.089276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa02e5-6713-4352-bbb8-82e3550683ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.109755 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9faa02e5-6713-4352-bbb8-82e3550683ba None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:10.110959 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9faa02e5-6713-4352-bbb8-82e3550683ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:10.120686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1390/2758] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:25:10.125518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e242c0-8a9e-460d-9f12-5b6f90327fae None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:10.137577 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e242c0-8a9e-460d-9f12-5b6f90327fae None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:10.142478 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:10.143089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.143373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.143645 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.144298 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:25:10.144625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:10.147285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:10.162548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1369/2759] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:25:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 01 04:25:10.167453 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa02e5-6713-4352-bbb8-82e3550683ba None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:10.182108 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa02e5-6713-4352-bbb8-82e3550683ba None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:10.187421 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:10.188079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.188498 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.188874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.189724 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:10.191785 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:10.222175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1391/2760] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:25:10 2022] POST /identity/v3/projects => generated 375 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:25:10.226591 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:10.238113 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3958c6a7-08f4-4cd0-9b55-d0f62c50fc08 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:10.293990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:10.294589 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.294865 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.295132 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.295758 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:10.297680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:10.320086 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1370/2761] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:25:10 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:25:10.324250 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:10.334844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e728a5-7f67-4045-9b48-b28563fb8b1a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:10.339869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:10.340462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.340816 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.341111 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.341798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:10.343442 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:10.353786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1392/2762] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:25:10.357623 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:10.368194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a128426-eecb-4f33-bfec-2f6bf361ec5f admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:10.372822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ab2b27-2406-4f4f-bc8e-8ab8b065967e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:10.373445 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ab2b27-2406-4f4f-bc8e-8ab8b065967e admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.373722 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ab2b27-2406-4f4f-bc8e-8ab8b065967e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.373991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ab2b27-2406-4f4f-bc8e-8ab8b065967e admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.374624 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ab2b27-2406-4f4f-bc8e-8ab8b065967e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:10.376183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ab2b27-2406-4f4f-bc8e-8ab8b065967e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:10.387270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1371/2763] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:25:10.391478 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:10.401692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9eb64-9e63-49ce-8028-071a1da5ee3b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:10.406248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:10.406889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.407165 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.407435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] PATH_INFO: `/v3/projects/a1c1e492262e44cab94a51e215413e61/users/9358a5806ca7428eaa75670d137b18eb/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.421941 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1c1e492262e44cab94a51e215413e61, user_id=9358a5806ca7428eaa75670d137b18eb, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:10.424675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:10.438220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1393/2764] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:25:10 2022] PUT /identity/v3/projects/a1c1e492262e44cab94a51e215413e61/users/9358a5806ca7428eaa75670d137b18eb/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:10.449349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.449634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.449910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.476677 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] MFA Rules not processed for user `9358a5806ca7428eaa75670d137b18eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:10.477911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:10.543057 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1372/2765] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:10 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:10.554860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:10.556531 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:10.616113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dee0af-f150-4a05-89f7-3772e901a5b2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:10.621443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35575aad-d6cd-498b-aee3-d0b213e55d6f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:10.622043 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35575aad-d6cd-498b-aee3-d0b213e55d6f service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:10.622323 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35575aad-d6cd-498b-aee3-d0b213e55d6f service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:10.622746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35575aad-d6cd-498b-aee3-d0b213e55d6f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:10.631276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35575aad-d6cd-498b-aee3-d0b213e55d6f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:10.632692 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35575aad-d6cd-498b-aee3-d0b213e55d6f service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:10.641210 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1394/2766] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:10 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:11.177874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:11.179602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:11.202103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:11.243769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d712848-c7d3-49f0-aa42-05b8495c66d4 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:11.247293 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:11.247888 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:11.248167 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:11.248436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:11.250190 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:11.312318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:11.314134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:11.325280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1373/2767] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:12.008793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028e3711-f86b-4449-bb21-55bf47fb1edb None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:12.009384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028e3711-f86b-4449-bb21-55bf47fb1edb None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:12.009384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028e3711-f86b-4449-bb21-55bf47fb1edb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:12.029880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-028e3711-f86b-4449-bb21-55bf47fb1edb None None] MFA Rules not processed for user `9358a5806ca7428eaa75670d137b18eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:12.031683 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-028e3711-f86b-4449-bb21-55bf47fb1edb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:12.042655 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1395/2768] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:14.746153 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:14.747770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:14.762691 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8a1739-c127-450c-8f3d-3757d377af70 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:14.768080 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:14.768719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:14.769036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:14.769317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] PATH_INFO: `/v3/users/ccd631925a0a4f5ca8be5d3915778487` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:14.770737 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccd631925a0a4f5ca8be5d3915778487)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:14.772685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:14.839474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccd631925a0a4f5ca8be5d3915778487'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:14.840206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccd631925a0a4f5ca8be5d3915778487'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:14.849328 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccd631925a0a4f5ca8be5d3915778487'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:14.850332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccd631925a0a4f5ca8be5d3915778487'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:14.863332 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-c72b1495-ec96-4b32-99f5-1c89810208b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccd631925a0a4f5ca8be5d3915778487'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:14.913802 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1374/2769] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:14 2022] DELETE /identity/v3/users/ccd631925a0a4f5ca8be5d3915778487 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:15.172797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028e3711-f86b-4449-bb21-55bf47fb1edb None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.187782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028e3711-f86b-4449-bb21-55bf47fb1edb None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:15.239052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:15.255663 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:15.256394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.256742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.257109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] PATH_INFO: `/v3/projects/19d87f6d9adf4ce98e934187ec8ae0d7` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.258327 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] RBAC: Authorizing `identity:delete_project(project_id=19d87f6d9adf4ce98e934187ec8ae0d7)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:15.260314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:15.309642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19d87f6d9adf4ce98e934187ec8ae0d7', 'request_id': 'req-0992ba42-3440-4aea-8f8d-94e09f92784f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:15.318462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19d87f6d9adf4ce98e934187ec8ae0d7', 'request_id': 'req-0992ba42-3440-4aea-8f8d-94e09f92784f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:15.325757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19d87f6d9adf4ce98e934187ec8ae0d7', 'request_id': 'req-0992ba42-3440-4aea-8f8d-94e09f92784f'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:15.339682 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] The token cache is being invalidate because project 19d87f6d9adf4ce98e934187ec8ae0d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:15.339974 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:15.340642 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-0992ba42-3440-4aea-8f8d-94e09f92784f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:15.342156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1396/2770] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:15 2022] DELETE /identity/v3/projects/19d87f6d9adf4ce98e934187ec8ae0d7 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:15.347053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95a94c-d452-4741-9751-f3d855fa8318 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.347335 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95a94c-d452-4741-9751-f3d855fa8318 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.347604 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95a94c-d452-4741-9751-f3d855fa8318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.366900 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-5a95a94c-d452-4741-9751-f3d855fa8318 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:15.368010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5a95a94c-d452-4741-9751-f3d855fa8318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:15.404356 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:15.432946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1375/2771] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:15.440806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058cf9e5-fce0-47d2-a8b8-c01d87db5f6d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.441196 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058cf9e5-fce0-47d2-a8b8-c01d87db5f6d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.441546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058cf9e5-fce0-47d2-a8b8-c01d87db5f6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.462592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-058cf9e5-fce0-47d2-a8b8-c01d87db5f6d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:15.463824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-058cf9e5-fce0-47d2-a8b8-c01d87db5f6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:15.472182 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1397/2772] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:15.476527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95a94c-d452-4741-9751-f3d855fa8318 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.487425 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95a94c-d452-4741-9751-f3d855fa8318 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:15.494040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:15.494660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.494949 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.495253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.495914 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:25:15.496223 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:15.497856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:15.512847 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1376/2773] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:25:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:25:15.517561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058cf9e5-fce0-47d2-a8b8-c01d87db5f6d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.528599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058cf9e5-fce0-47d2-a8b8-c01d87db5f6d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:15.534645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:15.536205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.536600 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.537274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.538219 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:15.540305 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:15.570919 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1398/2774] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:25:15 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:15.576259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.587036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a0375f-328e-4205-a0ea-8fd84840047a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:15.654965 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:15.655539 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.655874 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.656152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.657327 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:15.659391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:15.688548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1377/2775] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:25:15 2022] POST /identity/v3/users => generated 291 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:25:15.691130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.703904 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42519264-b1bb-40a7-a1d3-e35b5bf79a59 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:15.708647 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:15.709299 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.709593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.709879 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.710527 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:15.712143 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:15.722922 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1399/2776] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:15 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:25:15.727242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.741357 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab54439-6934-4600-bc44-3218035437a2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:15.746092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c93078-ffc9-48c0-9236-cb0de45a291d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:15.746700 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c93078-ffc9-48c0-9236-cb0de45a291d admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.747250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c93078-ffc9-48c0-9236-cb0de45a291d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.747250 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c93078-ffc9-48c0-9236-cb0de45a291d admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.747858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c93078-ffc9-48c0-9236-cb0de45a291d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:15.749492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c93078-ffc9-48c0-9236-cb0de45a291d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:15.766095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1378/2777] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:25:15 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:25:15.778316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.788944 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3196c1f9-7c5e-4811-bb5f-e184696fc815 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:15.793697 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:15.794725 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.795042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.795563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] PATH_INFO: `/v3/projects/57f927e7829a4172b9919b964a9e7c13/users/0e30d69a25494d70bc094eb92c4f9bb0/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.817732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=57f927e7829a4172b9919b964a9e7c13, user_id=0e30d69a25494d70bc094eb92c4f9bb0, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:15.821064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:15.851326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1400/2778] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:25:15 2022] PUT /identity/v3/projects/57f927e7829a4172b9919b964a9e7c13/users/0e30d69a25494d70bc094eb92c4f9bb0/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:25:15.856157 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457e429-8a86-45cd-b13b-311eed8e54ed None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:15.856458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457e429-8a86-45cd-b13b-311eed8e54ed None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:15.856719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457e429-8a86-45cd-b13b-311eed8e54ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:15.877750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2457e429-8a86-45cd-b13b-311eed8e54ed None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:15.879040 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2457e429-8a86-45cd-b13b-311eed8e54ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:15.941873 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1379/2779] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:15.952631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:15.954381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:16.014183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7036433f-8867-4ef0-b838-41ac0f8064bb admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:16.017800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:16.018420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:16.018701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:16.018978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:16.028899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:16.030433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:16.041378 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1401/2780] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:20.039710 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28903d-2a43-4f66-ac38-80e0dc6e5175 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:20.040172 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28903d-2a43-4f66-ac38-80e0dc6e5175 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:20.040291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af28903d-2a43-4f66-ac38-80e0dc6e5175 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:20.071141 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-af28903d-2a43-4f66-ac38-80e0dc6e5175 None None] MFA Rules not processed for user `0e30d69a25494d70bc094eb92c4f9bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:20.072373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-af28903d-2a43-4f66-ac38-80e0dc6e5175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:20.119867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:20.154642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1380/2781] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:20 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:20.169767 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:20.171756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:20.191749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:20.254529 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13190ea8-9e52-4067-a09c-146a644612e9 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:20.260535 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d0cea-1383-4a3f-8758-bf7ad3f1286b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:20.261176 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d0cea-1383-4a3f-8758-bf7ad3f1286b service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:20.261462 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d0cea-1383-4a3f-8758-bf7ad3f1286b service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:20.261727 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d0cea-1383-4a3f-8758-bf7ad3f1286b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:20.271193 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d0cea-1383-4a3f-8758-bf7ad3f1286b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:20.272843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d0cea-1383-4a3f-8758-bf7ad3f1286b service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:20.284565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1402/2782] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:20 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:22.018163 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:22.018719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:22.018719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:22.039260 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] MFA Rules not processed for user `0e30d69a25494d70bc094eb92c4f9bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:22.040520 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:22.047837 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1381/2783] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:22 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:25:27.909391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1764fd9-e772-4ae1-973a-963d27063c44 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:27.909868 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1764fd9-e772-4ae1-973a-963d27063c44 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:27.909966 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1764fd9-e772-4ae1-973a-963d27063c44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:27.930171 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f1764fd9-e772-4ae1-973a-963d27063c44 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:27.931400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f1764fd9-e772-4ae1-973a-963d27063c44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:27.985380 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1403/2784] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:25:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:28.033064 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:28.034486 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:28.103559 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66491814-b8b9-412c-9089-75bcad77a42a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:28.108488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:28.109119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:28.109404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:28.109680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:28.117951 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:28.119560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:28.127652 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1382/2785] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:25:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:25:28.626993 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30ecf8-dbb3-444a-9ce7-10304c4da49c None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:28.627499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30ecf8-dbb3-444a-9ce7-10304c4da49c None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:28.627777 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30ecf8-dbb3-444a-9ce7-10304c4da49c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:28.647069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-eb30ecf8-dbb3-444a-9ce7-10304c4da49c None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:28.648466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-eb30ecf8-dbb3-444a-9ce7-10304c4da49c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:28.655950 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1404/2786] 10.222.0.26 () {58 vars in 967 bytes} [Sat Oct 1 04:25:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:28.669620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:28.680959 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4fa83b-7fff-4aed-ab02-1b29e947c359 service swift] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:28.686036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880876e-c277-4332-9645-906151f2b51f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:28.686629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880876e-c277-4332-9645-906151f2b51f service swift] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:28.686908 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880876e-c277-4332-9645-906151f2b51f service swift] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:28.687180 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880876e-c277-4332-9645-906151f2b51f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:28.695617 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880876e-c277-4332-9645-906151f2b51f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:28.697218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880876e-c277-4332-9645-906151f2b51f service swift] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:28.705393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1383/2787] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:25:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:25:28.938710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8ad8ef-1272-4b93-8745-7c6644164fe1 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:28.939202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8ad8ef-1272-4b93-8745-7c6644164fe1 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:28.939420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8ad8ef-1272-4b93-8745-7c6644164fe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:28.959335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-ca8ad8ef-1272-4b93-8745-7c6644164fe1 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:28.960776 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-ca8ad8ef-1272-4b93-8745-7c6644164fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:28.968390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1405/2788] 10.222.0.26 () {58 vars in 1009 bytes} [Sat Oct 1 04:25:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:28.981676 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880876e-c277-4332-9645-906151f2b51f service swift] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:28.993527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880876e-c277-4332-9645-906151f2b51f service swift] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:28.998924 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:28.999575 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:28.999871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:29.000142 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:29.009313 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:29.010897 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:29.020409 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1384/2789] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:25:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:25:29.351284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:29.351754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:29.352253 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:29.371646 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] MFA Rules not processed for user `98e0eece1e7146c48cf68b599e58c0bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:29.372865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:29.381409 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1406/2790] 10.222.0.26 () {58 vars in 967 bytes} [Sat Oct 1 04:25:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:29.394634 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:29.405923 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12eb255c-96c9-43ea-9fd0-25d07f5e5e06 service swift] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:29.411029 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665934ac-1269-4d12-8478-f1d9c598c36a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3d69adfda7334567b03627a1d839f4f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:29.411632 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665934ac-1269-4d12-8478-f1d9c598c36a service swift] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:29.411917 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665934ac-1269-4d12-8478-f1d9c598c36a service swift] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:29.412194 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665934ac-1269-4d12-8478-f1d9c598c36a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:29.420034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665934ac-1269-4d12-8478-f1d9c598c36a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:29.421641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665934ac-1269-4d12-8478-f1d9c598c36a service swift] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:29.429491 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1385/2791] 10.222.0.26 () {60 vars in 1384 bytes} [Sat Oct 1 04:25:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 01 04:25:31.553649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:31.555028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:31.569197 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4af43db-9417-4abf-9d19-5481ba122e86 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:31.573604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:31.574220 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:31.574499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:31.574771 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] PATH_INFO: `/v3/users/9358a5806ca7428eaa75670d137b18eb` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:31.575883 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9358a5806ca7428eaa75670d137b18eb)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:31.577829 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:31.623441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9358a5806ca7428eaa75670d137b18eb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:31.631207 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9358a5806ca7428eaa75670d137b18eb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:31.631812 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9358a5806ca7428eaa75670d137b18eb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:31.632420 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9358a5806ca7428eaa75670d137b18eb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:31.639747 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9358a5806ca7428eaa75670d137b18eb'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:31.655223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:31.660921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1407/2792] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:25:31 2022] DELETE /identity/v3/users/9358a5806ca7428eaa75670d137b18eb => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:31.666091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:31.666580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:31.666679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:31.686262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:25:31.687696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:31.746743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1386/2793] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:25:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:25:31.756820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:31.758266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:31.815892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc518815-1909-454b-967f-14230ddfff41 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:31.819991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:31.820585 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:31.820862 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:31.821163 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:31.829969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:31.831388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:31.841206 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1408/2794] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:25:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:25:32.015647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:25:32.017092 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:25:32.030507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d9fb84-7daf-4b84-bd40-7104ed0fc2b6 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:25:32.036259 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:25:32.037195 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:25:32.037674 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:25:32.038081 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] PATH_INFO: `/v3/projects/a1c1e492262e44cab94a51e215413e61` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:25:32.039458 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1c1e492262e44cab94a51e215413e61)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:25:32.043188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:25:32.063496 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=100 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:32.094745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1c1e492262e44cab94a51e215413e61', 'request_id': 'req-5f0eebf7-07c2-4ddc-863e-27c59b371337'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:32.100487 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1c1e492262e44cab94a51e215413e61', 'request_id': 'req-5f0eebf7-07c2-4ddc-863e-27c59b371337'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:32.108641 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1c1e492262e44cab94a51e215413e61', 'request_id': 'req-5f0eebf7-07c2-4ddc-863e-27c59b371337'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:32.121252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] The token cache is being invalidate because project a1c1e492262e44cab94a51e215413e61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:25:32.121595 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:32.122274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5f0eebf7-07c2-4ddc-863e-27c59b371337 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:25:32.123760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1387/2795] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:25:32 2022] DELETE /identity/v3/projects/a1c1e492262e44cab94a51e215413e61 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:25:41.837101 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=26 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:25:42.117530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=18,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:26:05.103461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:26:05.104845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:26:05.188343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6e12d7-8e46-4f7f-b598-e45e8ce3a0c3 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:26:05.236986 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:26:05.237595 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:26:05.237880 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:26:05.238154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:26:05.240552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:26:05.311051 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:26:05.312518 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:26:05.321392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1409/2796] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:26:05 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:26:12.183266 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44426cff-d2d8-422e-b69b-69de08e755e7 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:26:12.183817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44426cff-d2d8-422e-b69b-69de08e755e7 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:26:12.183817 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44426cff-d2d8-422e-b69b-69de08e755e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:26:12.203105 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-44426cff-d2d8-422e-b69b-69de08e755e7 None None] MFA Rules not processed for user `dea6fd8e6a374cd5b12ec7d99f285203`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:26:12.204291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-44426cff-d2d8-422e-b69b-69de08e755e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:26:12.211693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1388/2797] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:26:12 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:26:13.441889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:26:13.443827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:26:13.476343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=100 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:26:13.542365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f505b330-14d2-49c1-89e0-6ad3ccdbadbb service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:26:13.547564 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6b79db-5af3-4a04-bf57-93051177afde service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:26:13.548252 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b79db-5af3-4a04-bf57-93051177afde service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:26:13.548567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b79db-5af3-4a04-bf57-93051177afde service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:26:13.548846 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6b79db-5af3-4a04-bf57-93051177afde service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:26:13.550677 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e6b79db-5af3-4a04-bf57-93051177afde service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:26:13.562629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6b79db-5af3-4a04-bf57-93051177afde service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:26:13.564430 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6b79db-5af3-4a04-bf57-93051177afde service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:26:13.575116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1410/2798] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:26:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:26:20.983114 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:26:20.983648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:26:20.983759 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:26:21.002980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] MFA Rules not processed for user `0e30d69a25494d70bc094eb92c4f9bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:26:21.004164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:26:21.057537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1389/2799] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:26:20 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:26:23.570754 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=31 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:26:31.050211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=28 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:27:20.736109 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e11297b-669d-4cf9-83cb-e27b30d74a37 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:27:20.736657 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e11297b-669d-4cf9-83cb-e27b30d74a37 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:27:20.736770 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e11297b-669d-4cf9-83cb-e27b30d74a37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:27:20.756277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-4e11297b-669d-4cf9-83cb-e27b30d74a37 None None] MFA Rules not processed for user `0e30d69a25494d70bc094eb92c4f9bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:27:20.757509 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-4e11297b-669d-4cf9-83cb-e27b30d74a37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:27:20.765162 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1411/2800] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:27:20 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:27:30.750736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:03.940252 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:03.941631 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:04.028770 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b28df6c-06e0-496d-8463-9c2ef3480c77 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:04.032642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:04.033280 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:04.033557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:04.033826 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:04.035547 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:04.125882 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:04.127312 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:04.149436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1390/2801] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:28:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:28:05.346979 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e11297b-669d-4cf9-83cb-e27b30d74a37 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:05.357266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e11297b-669d-4cf9-83cb-e27b30d74a37 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:05.361015 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:05.361622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:05.361934 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:05.362175 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:05.364032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:05.378648 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:05.380078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:05.390968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1412/2802] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:28:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:28:14.146809 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=75 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:15.386040 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:16.530354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:16.531821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:16.542530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71c6c03-ffe1-41ce-9b63-673ee3f1536c service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:16.546531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:16.547126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:16.547403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:16.547671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:16.549378 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:16.633753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:16.635310 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:16.657972 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1391/2803] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:28:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:28:19.928757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:19.931082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:19.960621 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53f66c9-a769-45fe-b738-bf3303ec7073 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:19.968622 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:19.969391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:19.970352 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:19.971422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:19.973602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:20.000276 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:20.002631 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:20.030289 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1413/2804] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:28:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:28:23.342529 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:23.344134 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:23.356735 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b436b88b-23fa-4c05-b887-98be4b16bc90 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:23.361749 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:23.362462 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:23.362852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:23.363346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] PATH_INFO: `/v3/users/dea6fd8e6a374cd5b12ec7d99f285203` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:23.364490 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dea6fd8e6a374cd5b12ec7d99f285203)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:23.366457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:23.415469 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dea6fd8e6a374cd5b12ec7d99f285203'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.416202 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dea6fd8e6a374cd5b12ec7d99f285203'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.423364 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dea6fd8e6a374cd5b12ec7d99f285203'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.424100 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dea6fd8e6a374cd5b12ec7d99f285203'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.431143 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6b19a72b-a6d2-4284-bcca-3a92b1e09390 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dea6fd8e6a374cd5b12ec7d99f285203'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.449435 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1392/2805] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:28:23 2022] DELETE /identity/v3/users/dea6fd8e6a374cd5b12ec7d99f285203 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:28:23.630844 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:23.648251 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fa73f8-9d31-4d32-9d8b-051b9e19f58f service nova] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:23.692275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:23.692760 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:23.693011 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:23.713407 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] MFA Rules not processed for user `0e30d69a25494d70bc094eb92c4f9bb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:28:23.714654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:23.723999 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:23.724639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:23.724918 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:23.725215 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] PATH_INFO: `/v3/projects/721dd52b82194e4d91113d4d688900ae` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:23.726318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] RBAC: Authorizing `identity:delete_project(project_id=721dd52b82194e4d91113d4d688900ae)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:23.728208 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:23.747585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:23.774899 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '721dd52b82194e4d91113d4d688900ae', 'request_id': 'req-835eb87c-e2ad-4f28-a930-c3f6be56714e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.776050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1393/2806] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:28:23 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:28:23.783406 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '721dd52b82194e4d91113d4d688900ae', 'request_id': 'req-835eb87c-e2ad-4f28-a930-c3f6be56714e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.790033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '721dd52b82194e4d91113d4d688900ae', 'request_id': 'req-835eb87c-e2ad-4f28-a930-c3f6be56714e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.801017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] The token cache is being invalidate because project 721dd52b82194e4d91113d4d688900ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:28:23.801316 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.801954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.803431 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1414/2807] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:28:23 2022] DELETE /identity/v3/projects/721dd52b82194e4d91113d4d688900ae => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:28:23.807494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:23.809044 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:23.822642 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd79d31a-5282-43e5-813d-d70afe394b59 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:23.881236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:23.881794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:23.882193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:23.882485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] PATH_INFO: `/v3/users/392fdfae893e492fa9e08cc870b714a5` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:23.883614 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=392fdfae893e492fa9e08cc870b714a5)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:23.885600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:23.932738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '392fdfae893e492fa9e08cc870b714a5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.933428 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '392fdfae893e492fa9e08cc870b714a5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.940290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '392fdfae893e492fa9e08cc870b714a5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.940966 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '392fdfae893e492fa9e08cc870b714a5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.948112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '392fdfae893e492fa9e08cc870b714a5'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:23.968548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1394/2808] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:28:23 2022] DELETE /identity/v3/users/392fdfae893e492fa9e08cc870b714a5 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:28:24.146633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:24.148281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:24.165445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835eb87c-e2ad-4f28-a930-c3f6be56714e admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:24.229001 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:24.229744 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:24.230029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:24.230309 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] PATH_INFO: `/v3/projects/997939da104a452fba57b826ccae88d7` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:24.232128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=997939da104a452fba57b826ccae88d7)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:24.234130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:24.255828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:24.282970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '997939da104a452fba57b826ccae88d7', 'request_id': 'req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:24.289656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '997939da104a452fba57b826ccae88d7', 'request_id': 'req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:24.296281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '997939da104a452fba57b826ccae88d7', 'request_id': 'req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:24.306954 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] The token cache is being invalidate because project 997939da104a452fba57b826ccae88d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:28:24.307343 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:24.308026 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:24.310296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1415/2809] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:28:24 2022] DELETE /identity/v3/projects/997939da104a452fba57b826ccae88d7 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:28:33.964268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=61,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:34.303850 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:54.762667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:54.764237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:54.777272 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d6840a-1178-4702-8000-26c1fd5906e1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:54.833302 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:54.833928 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:54.834206 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:54.834476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] PATH_INFO: `/v3/users/944a3ab108184d5ab8784e21805f9873` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:54.835572 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=944a3ab108184d5ab8784e21805f9873)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:54.837537 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:54.883532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '944a3ab108184d5ab8784e21805f9873'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:54.884178 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '944a3ab108184d5ab8784e21805f9873'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:54.891344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '944a3ab108184d5ab8784e21805f9873'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:54.891980 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '944a3ab108184d5ab8784e21805f9873'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:54.900014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '944a3ab108184d5ab8784e21805f9873'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:54.917783 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1395/2810] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:28:54 2022] DELETE /identity/v3/users/944a3ab108184d5ab8784e21805f9873 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:28:55.090903 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:55.093267 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:55.106434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6391f9-1744-42b1-b7b0-93ba8cf0d3d2 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:55.164741 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:55.165411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:55.165695 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:55.165972 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] PATH_INFO: `/v3/projects/1eda12202e6f439b96a87a81d7531503` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:55.167090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=1eda12202e6f439b96a87a81d7531503)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:55.169836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:55.215456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1eda12202e6f439b96a87a81d7531503', 'request_id': 'req-d7505901-7341-4826-beb7-e851512ae3ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.223199 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1eda12202e6f439b96a87a81d7531503', 'request_id': 'req-d7505901-7341-4826-beb7-e851512ae3ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.229456 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1eda12202e6f439b96a87a81d7531503', 'request_id': 'req-d7505901-7341-4826-beb7-e851512ae3ab'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.239772 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] The token cache is being invalidate because project 1eda12202e6f439b96a87a81d7531503 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:28:55.240049 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.240762 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.242303 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1416/2811] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:28:55 2022] DELETE /identity/v3/projects/1eda12202e6f439b96a87a81d7531503 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:28:55.246474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:55.247849 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:55.260856 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74812b20-6f38-433e-a86c-1aba33c357e8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:55.316097 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:55.316725 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:55.317039 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:55.317319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] PATH_INFO: `/v3/users/edf10e00650444f6b379d694d80f771d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:55.318422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=edf10e00650444f6b379d694d80f771d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:55.320325 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:55.342567 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:55.369789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'edf10e00650444f6b379d694d80f771d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.370467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'edf10e00650444f6b379d694d80f771d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.382377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'edf10e00650444f6b379d694d80f771d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.383405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'edf10e00650444f6b379d694d80f771d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.391181 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'edf10e00650444f6b379d694d80f771d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.409427 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1396/2812] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:28:55 2022] DELETE /identity/v3/users/edf10e00650444f6b379d694d80f771d => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:28:55.578539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:28:55.579995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:28:55.593164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7505901-7341-4826-beb7-e851512ae3ab admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:28:55.650540 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:28:55.651168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:28:55.651444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:28:55.651717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] PATH_INFO: `/v3/projects/2dccf74ffdb5456e88c122f02cb84f0f` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:28:55.652818 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2dccf74ffdb5456e88c122f02cb84f0f)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:28:55.654889 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:28:55.689522 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:28:55.703836 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2dccf74ffdb5456e88c122f02cb84f0f', 'request_id': 'req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.711472 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2dccf74ffdb5456e88c122f02cb84f0f', 'request_id': 'req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.717757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2dccf74ffdb5456e88c122f02cb84f0f', 'request_id': 'req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.727686 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] The token cache is being invalidate because project 2dccf74ffdb5456e88c122f02cb84f0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:28:55.727976 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.728599 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:28:55.730333 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1417/2813] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:28:55 2022] DELETE /identity/v3/projects/2dccf74ffdb5456e88c122f02cb84f0f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:05.405932 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=7,keystone:SELECT=20,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:05.724758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:29.621974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:29.623459 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:29.641244 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e09385-6e65-4c4b-a6c3-43b0a6f19cb3 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:29.696309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:29.696986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:29.697239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:29.697513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] PATH_INFO: `/v3/users/0e30d69a25494d70bc094eb92c4f9bb0` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:29.698636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e30d69a25494d70bc094eb92c4f9bb0)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:29.701253 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:29.747177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e30d69a25494d70bc094eb92c4f9bb0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:29.747910 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e30d69a25494d70bc094eb92c4f9bb0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:29.754683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e30d69a25494d70bc094eb92c4f9bb0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:29.755291 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e30d69a25494d70bc094eb92c4f9bb0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:29.761733 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-ea19c4ee-9da6-4efe-9296-f15b4aa4bfc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e30d69a25494d70bc094eb92c4f9bb0'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:29.779165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1397/2814] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:29:29 2022] DELETE /identity/v3/users/0e30d69a25494d70bc094eb92c4f9bb0 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:29:29.954330 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:29.955795 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:29.968353 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9593ff1d-2bd3-41d4-94ab-22ad6d01f2c1 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.028464 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.029264 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.029567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.029842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] PATH_INFO: `/v3/projects/57f927e7829a4172b9919b964a9e7c13` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.030957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57f927e7829a4172b9919b964a9e7c13)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.032865 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.079434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57f927e7829a4172b9919b964a9e7c13', 'request_id': 'req-56d87de8-6639-4aca-a087-2e75dd8f6019'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:30.086700 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57f927e7829a4172b9919b964a9e7c13', 'request_id': 'req-56d87de8-6639-4aca-a087-2e75dd8f6019'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:30.094052 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57f927e7829a4172b9919b964a9e7c13', 'request_id': 'req-56d87de8-6639-4aca-a087-2e75dd8f6019'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:30.105486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] The token cache is being invalidate because project 57f927e7829a4172b9919b964a9e7c13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:29:30.105780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:30.107028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-56d87de8-6639-4aca-a087-2e75dd8f6019 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:30.108589 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1418/2815] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:29:29 2022] DELETE /identity/v3/projects/57f927e7829a4172b9919b964a9e7c13 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:30.113507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dec71b-677c-4ae4-afa6-f95076fceaa9 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.113867 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dec71b-677c-4ae4-afa6-f95076fceaa9 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.114174 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dec71b-677c-4ae4-afa6-f95076fceaa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.174472 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-69dec71b-677c-4ae4-afa6-f95076fceaa9 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:30.175697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-69dec71b-677c-4ae4-afa6-f95076fceaa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:30.219309 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:30.236483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1398/2816] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:30.244510 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2906a0-7861-4335-8cbe-997003fc780d None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.244797 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2906a0-7861-4335-8cbe-997003fc780d None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.245128 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2906a0-7861-4335-8cbe-997003fc780d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.265471 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-0e2906a0-7861-4335-8cbe-997003fc780d None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:30.266681 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-0e2906a0-7861-4335-8cbe-997003fc780d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:30.274791 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1419/2817] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:30.279091 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dec71b-677c-4ae4-afa6-f95076fceaa9 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.288745 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dec71b-677c-4ae4-afa6-f95076fceaa9 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.294124 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.294988 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.295483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.295795 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.296444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:29:30.296753 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.298373 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.310476 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1399/2818] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:29:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:29:30.314680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2906a0-7861-4335-8cbe-997003fc780d None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.324014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2906a0-7861-4335-8cbe-997003fc780d None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.328987 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.329680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.329964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.330237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.330859 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.332735 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.369619 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1420/2819] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:29:30 2022] POST /identity/v3/projects => generated 367 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:29:30.374390 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.384835 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f61816d-054c-4505-8599-c8797bd3cd5b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.443380 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.444006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.444284 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.444556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.445300 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.447224 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.469696 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1400/2820] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:29:30 2022] POST /identity/v3/users => generated 284 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:29:30.473834 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.485130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d854db-2b99-474f-b9d6-59f679fd9df4 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.489827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.490423 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.490701 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.490973 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.491604 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.493216 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.503989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1421/2821] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:29:30.508034 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.519099 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2085af9d-3044-47a1-8a1d-993320d3f47b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.523830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d69a80-4622-457e-817e-c68c47a69507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.524456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d69a80-4622-457e-817e-c68c47a69507 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.524736 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d69a80-4622-457e-817e-c68c47a69507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.525036 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d69a80-4622-457e-817e-c68c47a69507 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.525679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d69a80-4622-457e-817e-c68c47a69507 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.527344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d69a80-4622-457e-817e-c68c47a69507 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.538236 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1401/2822] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:29:30.542328 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.552574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef8153b-7581-4803-88d8-23bdc3713dfd admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.557168 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.557809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.558090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.558366 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] PATH_INFO: `/v3/projects/6ec3f586a8044d1ea4d532e957ac4c4e/users/82a1b6a4db32445a9c5bcf45dc29d1cd/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.573571 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ec3f586a8044d1ea4d532e957ac4c4e, user_id=82a1b6a4db32445a9c5bcf45dc29d1cd, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.577180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.589371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1422/2823] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:29:30 2022] PUT /identity/v3/projects/6ec3f586a8044d1ea4d532e957ac4c4e/users/82a1b6a4db32445a9c5bcf45dc29d1cd/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:29:30.599793 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b8cbdb-9524-4d5e-8aa6-2a4641453bb3 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.600095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b8cbdb-9524-4d5e-8aa6-2a4641453bb3 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.600351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b8cbdb-9524-4d5e-8aa6-2a4641453bb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.627548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-76b8cbdb-9524-4d5e-8aa6-2a4641453bb3 None None] MFA Rules not processed for user `82a1b6a4db32445a9c5bcf45dc29d1cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:30.628731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-76b8cbdb-9524-4d5e-8aa6-2a4641453bb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:30.692574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1402/2824] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:30 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:29:30.697142 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.716710 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:30.761057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da01c8a4-f5c5-497b-a404-ad2968ab6eb6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.766290 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.766885 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.767161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.767432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.768340 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.770326 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.796432 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1423/2825] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:29:30 2022] POST /identity/v3/projects => generated 367 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:29:30.800613 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b8cbdb-9524-4d5e-8aa6-2a4641453bb3 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.811616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b8cbdb-9524-4d5e-8aa6-2a4641453bb3 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.850935 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:30.867500 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.868126 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.868405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.868672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.869346 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.871234 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.891941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1403/2826] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:29:30 2022] POST /identity/v3/users => generated 290 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:29:30.896157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.906376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7353ff37-e331-46bb-ab40-7f5b7f7efc66 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.910877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.911473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.911752 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.912022 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.912656 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.914269 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.925358 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1424/2827] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 01 04:29:30.929301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.939683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b30a246-2f46-447f-9ecf-a5f66234d536 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:30.944262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:30.944842 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:30.945144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:30.945416 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] PATH_INFO: `/v3/projects/10849b036f524fc994cf92b2fa3dc2c5/users/52da4c228e72455eaf43b2a4f789fe36/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:30.959921 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=10849b036f524fc994cf92b2fa3dc2c5, user_id=52da4c228e72455eaf43b2a4f789fe36, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:30.962570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:30.974824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1404/2828] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:29:30 2022] PUT /identity/v3/projects/10849b036f524fc994cf92b2fa3dc2c5/users/52da4c228e72455eaf43b2a4f789fe36/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:30.978736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:30.989494 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0f2c19-af4c-45b7-b66a-83712e5e4bcc admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:31.046029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58b8516-67b7-4c98-95d8-d029b6019006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:31.046630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b8516-67b7-4c98-95d8-d029b6019006 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:31.046907 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b8516-67b7-4c98-95d8-d029b6019006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:31.047178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b8516-67b7-4c98-95d8-d029b6019006 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:31.047833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58b8516-67b7-4c98-95d8-d029b6019006 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:31.049450 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58b8516-67b7-4c98-95d8-d029b6019006 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:31.058983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1425/2829] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:30 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:29:31.064003 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:31.074177 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8937131c-5ec9-4414-ab3b-7aefa3c050bb admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:31.078915 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:31.079563 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:31.079846 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:31.080121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] PATH_INFO: `/v3/projects/10849b036f524fc994cf92b2fa3dc2c5/users/52da4c228e72455eaf43b2a4f789fe36/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:31.082072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=10849b036f524fc994cf92b2fa3dc2c5, user_id=52da4c228e72455eaf43b2a4f789fe36, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:31.084683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:31.094970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1405/2830] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:29:31 2022] PUT /identity/v3/projects/10849b036f524fc994cf92b2fa3dc2c5/users/52da4c228e72455eaf43b2a4f789fe36/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:31.107275 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9274a2ac-d4c1-4802-9cb0-5cf87a8488b0 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:31.107605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9274a2ac-d4c1-4802-9cb0-5cf87a8488b0 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:31.107884 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9274a2ac-d4c1-4802-9cb0-5cf87a8488b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:31.135983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-9274a2ac-d4c1-4802-9cb0-5cf87a8488b0 None None] MFA Rules not processed for user `52da4c228e72455eaf43b2a4f789fe36`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:31.137226 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-9274a2ac-d4c1-4802-9cb0-5cf87a8488b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:31.204150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1426/2831] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:31 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:31.214718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:31.217548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:31.302992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67175c69-e620-4c48-b00e-e77e9cfeb7a5 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:31.307686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75121966-d15b-481d-b50c-7e3c0e5f2137 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:31.308282 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75121966-d15b-481d-b50c-7e3c0e5f2137 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:31.308566 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75121966-d15b-481d-b50c-7e3c0e5f2137 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:31.308848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75121966-d15b-481d-b50c-7e3c0e5f2137 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:31.317813 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75121966-d15b-481d-b50c-7e3c0e5f2137 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:31.319228 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75121966-d15b-481d-b50c-7e3c0e5f2137 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:31.329413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1406/2832] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:29:31 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:29:32.664575 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9274a2ac-d4c1-4802-9cb0-5cf87a8488b0 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:32.677929 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:32.726523 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9274a2ac-d4c1-4802-9cb0-5cf87a8488b0 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:32.730951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:32.732161 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:32.732438 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:32.732707 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] PATH_INFO: `/v3/users/82a1b6a4db32445a9c5bcf45dc29d1cd` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:32.733838 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] RBAC: Authorizing `identity:delete_user(user_id=82a1b6a4db32445a9c5bcf45dc29d1cd)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:32.735736 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:32.780633 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82a1b6a4db32445a9c5bcf45dc29d1cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:32.787363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82a1b6a4db32445a9c5bcf45dc29d1cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:32.788017 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82a1b6a4db32445a9c5bcf45dc29d1cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:32.788623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82a1b6a4db32445a9c5bcf45dc29d1cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:32.795335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82a1b6a4db32445a9c5bcf45dc29d1cd'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:32.812546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1427/2833] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:29:32 2022] DELETE /identity/v3/users/82a1b6a4db32445a9c5bcf45dc29d1cd => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:32.817211 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:32.817697 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:32.817799 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:32.836823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:32.838033 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:32.895585 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1407/2834] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:32.905556 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:32.906963 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:32.964614 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6273d01-1d5d-4ab7-bfb8-3a7fba87057b admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:32.968266 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:32.968869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:32.969178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:32.969449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:32.978201 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:32.979626 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:32.988821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1428/2835] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:29:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:29:33.251858 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:33.253404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:33.266643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb886e7-606e-41af-b4ec-76d52d3877f8 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:33.271780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:33.272354 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:33.272635 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:33.272905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] PATH_INFO: `/v3/projects/6ec3f586a8044d1ea4d532e957ac4c4e` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:33.274043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ec3f586a8044d1ea4d532e957ac4c4e)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:33.275936 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:33.292562 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:33.324492 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ec3f586a8044d1ea4d532e957ac4c4e', 'request_id': 'req-5d080ecb-5050-48ae-8787-e519d9d4486d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.330473 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ec3f586a8044d1ea4d532e957ac4c4e', 'request_id': 'req-5d080ecb-5050-48ae-8787-e519d9d4486d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.337573 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ec3f586a8044d1ea4d532e957ac4c4e', 'request_id': 'req-5d080ecb-5050-48ae-8787-e519d9d4486d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.349103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] The token cache is being invalidate because project 6ec3f586a8044d1ea4d532e957ac4c4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:29:33.349395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.350026 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.351531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1408/2836] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:29:33 2022] DELETE /identity/v3/projects/6ec3f586a8044d1ea4d532e957ac4c4e => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:33.355673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:33.365157 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89730fe8-063f-4970-b615-c344c3dffa7f service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:33.397499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:33.424519 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:33.425179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:33.425458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:33.425817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] PATH_INFO: `/v3/users/52da4c228e72455eaf43b2a4f789fe36` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:33.426877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] RBAC: Authorizing `identity:delete_user(user_id=52da4c228e72455eaf43b2a4f789fe36)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:33.428778 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:33.474651 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52da4c228e72455eaf43b2a4f789fe36'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.482678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52da4c228e72455eaf43b2a4f789fe36'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.483342 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52da4c228e72455eaf43b2a4f789fe36'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.483977 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52da4c228e72455eaf43b2a4f789fe36'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.491113 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-26dfcd48-734d-48e8-922d-90dbff68e90f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52da4c228e72455eaf43b2a4f789fe36'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.507953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1429/2837] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:29:33 2022] DELETE /identity/v3/users/52da4c228e72455eaf43b2a4f789fe36 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:33.785329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:33.786895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:33.802022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d080ecb-5050-48ae-8787-e519d9d4486d admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:33.858643 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:33.859283 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:33.859488 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:33.859764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] PATH_INFO: `/v3/projects/10849b036f524fc994cf92b2fa3dc2c5` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:33.860899 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=10849b036f524fc994cf92b2fa3dc2c5)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:33.862824 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:33.909329 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '10849b036f524fc994cf92b2fa3dc2c5', 'request_id': 'req-00f57b9f-7d91-431c-ace7-992b1a86df28'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.914743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '10849b036f524fc994cf92b2fa3dc2c5', 'request_id': 'req-00f57b9f-7d91-431c-ace7-992b1a86df28'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.922022 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '10849b036f524fc994cf92b2fa3dc2c5', 'request_id': 'req-00f57b9f-7d91-431c-ace7-992b1a86df28'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.932864 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] The token cache is being invalidate because project 10849b036f524fc994cf92b2fa3dc2c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:29:33.933165 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.933789 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-00f57b9f-7d91-431c-ace7-992b1a86df28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:29:33.935344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1409/2838] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:29:33 2022] DELETE /identity/v3/projects/10849b036f524fc994cf92b2fa3dc2c5 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:33.940308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ae61aa-8f89-4943-be97-550be0b4422b None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:33.940793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ae61aa-8f89-4943-be97-550be0b4422b None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:33.940902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ae61aa-8f89-4943-be97-550be0b4422b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:33.960733 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-a3ae61aa-8f89-4943-be97-550be0b4422b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:33.961970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-a3ae61aa-8f89-4943-be97-550be0b4422b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:34.020780 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1430/2839] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:34.030262 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df62230-b096-40e1-9f9d-428d164c2095 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.030773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df62230-b096-40e1-9f9d-428d164c2095 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.031281 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df62230-b096-40e1-9f9d-428d164c2095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.050976 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-6df62230-b096-40e1-9f9d-428d164c2095 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:34.052170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-6df62230-b096-40e1-9f9d-428d164c2095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:34.060237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1410/2840] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:34.065746 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ae61aa-8f89-4943-be97-550be0b4422b None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.077257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ae61aa-8f89-4943-be97-550be0b4422b None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.081935 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.082534 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.082814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.083090 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.083739 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:29:34.084044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.085680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.099623 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1431/2841] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:29:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:29:34.102408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df62230-b096-40e1-9f9d-428d164c2095 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.111821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df62230-b096-40e1-9f9d-428d164c2095 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.116493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.117127 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.117413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.117692 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.118376 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.120268 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.149690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1411/2842] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:29:34 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:29:34.155002 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.164774 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b561f4e-0f3b-464d-a558-c412a92a994d admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.205565 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:34.223189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.223793 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.224073 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.224357 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.225014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.226953 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.250499 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1432/2843] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:29:34 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:29:34.254687 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.261318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:34.266620 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553d999c-8bca-4e77-b78a-7b93c7e27def admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.272532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.273214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.273698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.274009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.274671 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.276274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.286801 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1412/2844] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:29:34.290723 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.301148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ee495c-35b0-4385-9967-c347c568674c admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.305708 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fe987f-e3f4-4d6d-9dd1-4859c74d50e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.306301 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fe987f-e3f4-4d6d-9dd1-4859c74d50e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.306580 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fe987f-e3f4-4d6d-9dd1-4859c74d50e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.306854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fe987f-e3f4-4d6d-9dd1-4859c74d50e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.307486 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fe987f-e3f4-4d6d-9dd1-4859c74d50e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.309120 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fe987f-e3f4-4d6d-9dd1-4859c74d50e7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.319503 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1433/2845] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:29:34.323456 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.333391 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044da2d3-8afe-44b9-94c6-c16b966fb612 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.337974 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.338608 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.338884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.339152 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] PATH_INFO: `/v3/projects/82166a1328ea43c9bd8d8618e36da9dd/users/85431dc72abb493894802de39c09be8e/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.353663 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] RBAC: Authorizing `identity:create_grant(project_id=82166a1328ea43c9bd8d8618e36da9dd, user_id=85431dc72abb493894802de39c09be8e, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.356911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.370587 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1413/2846] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:29:34 2022] PUT /identity/v3/projects/82166a1328ea43c9bd8d8618e36da9dd/users/85431dc72abb493894802de39c09be8e/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:34.382225 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed2035-2b8e-489a-850c-335853b163e2 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.382514 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed2035-2b8e-489a-850c-335853b163e2 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.382789 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ed2035-2b8e-489a-850c-335853b163e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.410995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f9ed2035-2b8e-489a-850c-335853b163e2 None None] MFA Rules not processed for user `85431dc72abb493894802de39c09be8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:34.412205 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f9ed2035-2b8e-489a-850c-335853b163e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:34.475256 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1434/2847] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:34 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:34.479868 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.541344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2367273-fd65-442f-936b-82ece77e825a admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.548351 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.549404 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.549698 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.549977 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.550618 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.552574 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.582279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1414/2848] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:29:34 2022] POST /identity/v3/projects => generated 377 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:29:34.586493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ed2035-2b8e-489a-850c-335853b163e2 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.597704 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ed2035-2b8e-489a-850c-335853b163e2 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.653315 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.653839 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.654115 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.654403 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.655014 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.656917 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.678284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1435/2849] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:29:34 2022] POST /identity/v3/users => generated 295 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:29:34.682279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.693043 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f92257-ae01-46b5-9611-32fd9b6e75e1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.697680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.698273 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.698556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.698830 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.699474 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.701340 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.711731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1415/2850] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:29:34.715998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.725731 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b48f4c-7f0f-480e-bbf2-5039c1e41c3a admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.730307 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.731579 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.731757 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.732033 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] PATH_INFO: `/v3/projects/dddff7c75e5f4e82a1277952a451f665/users/df3b42e62ce34b6f8f94a8d39626e193/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.746443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dddff7c75e5f4e82a1277952a451f665, user_id=df3b42e62ce34b6f8f94a8d39626e193, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.749306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.761806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1436/2851] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:29:34 2022] PUT /identity/v3/projects/dddff7c75e5f4e82a1277952a451f665/users/df3b42e62ce34b6f8f94a8d39626e193/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:34.766225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.776296 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09192e52-1809-4429-aa13-ae76767d9e21 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.831278 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321d3531-0f9c-46da-8a02-96477bbc96f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.831887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321d3531-0f9c-46da-8a02-96477bbc96f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.832171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321d3531-0f9c-46da-8a02-96477bbc96f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.832448 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321d3531-0f9c-46da-8a02-96477bbc96f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.833119 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321d3531-0f9c-46da-8a02-96477bbc96f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.834693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321d3531-0f9c-46da-8a02-96477bbc96f6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.845161 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1416/2852] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:29:34 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:29:34.849336 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:34.859548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6fa709-ec92-4dc1-bb19-060d00f888c7 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:34.864164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:34.864809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.865118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.865395 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] PATH_INFO: `/v3/projects/dddff7c75e5f4e82a1277952a451f665/users/df3b42e62ce34b6f8f94a8d39626e193/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.867306 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dddff7c75e5f4e82a1277952a451f665, user_id=df3b42e62ce34b6f8f94a8d39626e193, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:34.869969 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:34.880132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1437/2853] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:29:34 2022] PUT /identity/v3/projects/dddff7c75e5f4e82a1277952a451f665/users/df3b42e62ce34b6f8f94a8d39626e193/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:29:34.890794 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7080ba-b840-42b1-90b5-1192c7b766c1 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:34.891275 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7080ba-b840-42b1-90b5-1192c7b766c1 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:34.891395 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7080ba-b840-42b1-90b5-1192c7b766c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:34.919319 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-2e7080ba-b840-42b1-90b5-1192c7b766c1 None None] MFA Rules not processed for user `df3b42e62ce34b6f8f94a8d39626e193`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:29:34.920470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-2e7080ba-b840-42b1-90b5-1192c7b766c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:34.949000 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:34.988859 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1417/2854] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:29:34 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:29:35.000390 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:35.002221 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:29:35.007537 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:35.061629 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd7048-c8c2-41da-bbd6-cdad933cbf17 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:35.065318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:35.065921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:35.066202 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:35.066479 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:35.121080 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:35.123134 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:35.131726 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1438/2855] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:29:34 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:29:39.750379 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7080ba-b840-42b1-90b5-1192c7b766c1 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:29:39.759829 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7080ba-b840-42b1-90b5-1192c7b766c1 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:29:39.763654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1fcf8e-ac10-40f7-b315-cc4d9bf76161 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:29:39.764246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1fcf8e-ac10-40f7-b315-cc4d9bf76161 service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:29:39.764531 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1fcf8e-ac10-40f7-b315-cc4d9bf76161 service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:29:39.764804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1fcf8e-ac10-40f7-b315-cc4d9bf76161 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:29:39.773818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1fcf8e-ac10-40f7-b315-cc4d9bf76161 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:29:39.775176 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1fcf8e-ac10-40f7-b315-cc4d9bf76161 service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:29:39.785053 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1418/2856] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:29:39 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:29:45.127821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=47 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:29:49.780970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=22 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:02.760546 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:02.821938 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1652468-0a90-461b-b37d-9c2025118527 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:02.826416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:02.827036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:02.827318 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:02.827592 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] PATH_INFO: `/v3/users/85431dc72abb493894802de39c09be8e` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:02.828702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] RBAC: Authorizing `identity:delete_user(user_id=85431dc72abb493894802de39c09be8e)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:02.831454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:02.875057 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85431dc72abb493894802de39c09be8e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:02.882341 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85431dc72abb493894802de39c09be8e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:02.883029 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85431dc72abb493894802de39c09be8e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:02.883632 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85431dc72abb493894802de39c09be8e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:02.890769 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85431dc72abb493894802de39c09be8e'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:02.909576 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1439/2857] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:30:02 2022] DELETE /identity/v3/users/85431dc72abb493894802de39c09be8e => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:02.914083 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:02.914580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:02.914688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:02.935208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:02.936415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:02.995138 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1419/2858] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:03.005702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:03.007098 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:03.063154 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259595a-7c75-4ab0-8324-36a19b4c96af admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:03.067665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:03.068263 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:03.068541 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:03.068814 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:03.077572 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:03.078978 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:03.088753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1440/2859] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:03.355254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:03.356667 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:03.368553 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ad86d-2a4c-4376-b84b-b2360476e73b None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:03.373522 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:03.374160 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:03.374433 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:03.374738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] PATH_INFO: `/v3/projects/82166a1328ea43c9bd8d8618e36da9dd` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:03.375852 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82166a1328ea43c9bd8d8618e36da9dd)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:03.378507 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:03.424792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82166a1328ea43c9bd8d8618e36da9dd', 'request_id': 'req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.430246 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82166a1328ea43c9bd8d8618e36da9dd', 'request_id': 'req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.437483 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82166a1328ea43c9bd8d8618e36da9dd', 'request_id': 'req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.449269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] The token cache is being invalidate because project 82166a1328ea43c9bd8d8618e36da9dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:30:03.449564 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.450208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.451693 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1420/2860] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:30:03 2022] DELETE /identity/v3/projects/82166a1328ea43c9bd8d8618e36da9dd => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:03.456448 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:03.466141 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0fea6-6ad1-4a58-9d9c-988b693042a5 service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:03.483561 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:03.523763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:03.524401 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:03.524678 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:03.524996 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] PATH_INFO: `/v3/users/df3b42e62ce34b6f8f94a8d39626e193` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:03.526178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df3b42e62ce34b6f8f94a8d39626e193)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:03.528076 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:03.574130 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df3b42e62ce34b6f8f94a8d39626e193'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.581520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df3b42e62ce34b6f8f94a8d39626e193'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.582169 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df3b42e62ce34b6f8f94a8d39626e193'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.582759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df3b42e62ce34b6f8f94a8d39626e193'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.589970 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-2f7da77a-67d5-4d9c-8210-35b7bb7bc073 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df3b42e62ce34b6f8f94a8d39626e193'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:03.607612 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1441/2861] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:30:03 2022] DELETE /identity/v3/users/df3b42e62ce34b6f8f94a8d39626e193 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:03.879475 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:03.881074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:03.895785 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452dbb50-49ab-4c21-9ce2-42b5f6aeb3c6 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:03.954571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:03.955225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:03.955468 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:03.955743 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] PATH_INFO: `/v3/projects/dddff7c75e5f4e82a1277952a451f665` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:03.956866 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dddff7c75e5f4e82a1277952a451f665)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:03.958800 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:03.988393 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:04.006877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dddff7c75e5f4e82a1277952a451f665', 'request_id': 'req-6517ee37-3553-48de-9bfa-ffb1ee235b69'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:04.012265 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dddff7c75e5f4e82a1277952a451f665', 'request_id': 'req-6517ee37-3553-48de-9bfa-ffb1ee235b69'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:04.019651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dddff7c75e5f4e82a1277952a451f665', 'request_id': 'req-6517ee37-3553-48de-9bfa-ffb1ee235b69'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:04.030362 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] The token cache is being invalidate because project dddff7c75e5f4e82a1277952a451f665 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81586) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:30:04.030650 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:04.031279 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-6517ee37-3553-48de-9bfa-ffb1ee235b69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:04.033196 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1421/2862] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:30:03 2022] DELETE /identity/v3/projects/dddff7c75e5f4e82a1277952a451f665 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:04.038230 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a60bc-1c93-43ac-8931-dcd697d4d5b9 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.038712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a60bc-1c93-43ac-8931-dcd697d4d5b9 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.038920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7a60bc-1c93-43ac-8931-dcd697d4d5b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.059248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-cd7a60bc-1c93-43ac-8931-dcd697d4d5b9 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:04.060473 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cd7a60bc-1c93-43ac-8931-dcd697d4d5b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:04.119662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1442/2863] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:04.128791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03975b29-b192-4290-bc97-beb5168ca3c6 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.129108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03975b29-b192-4290-bc97-beb5168ca3c6 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.129386 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03975b29-b192-4290-bc97-beb5168ca3c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.149218 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-03975b29-b192-4290-bc97-beb5168ca3c6 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:04.150415 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-03975b29-b192-4290-bc97-beb5168ca3c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:04.158828 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1422/2864] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:04.163064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7a60bc-1c93-43ac-8931-dcd697d4d5b9 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.172533 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7a60bc-1c93-43ac-8931-dcd697d4d5b9 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.177702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.178292 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.178568 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.178837 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] PATH_INFO: `/v3/domains` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.179466 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81587) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:30:04.179763 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.181365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.193355 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1443/2865] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:30:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:30:04.198274 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03975b29-b192-4290-bc97-beb5168ca3c6 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.208248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03975b29-b192-4290-bc97-beb5168ca3c6 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.213009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.213612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.213895 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.214201 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.214843 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.216765 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.244237 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1423/2866] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:30:04.249513 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.259809 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72733acf-67c6-4736-a10b-d4b696943dcf admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.285860 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:04.318517 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.319118 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.319402 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.319680 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.320319 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.322262 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.345376 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1444/2867] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:30:04.349467 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.359592 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f703f25b-ca7b-4fdd-b07b-59032c1f566e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.364857 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.365485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.365769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.366046 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.366680 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.368290 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.378719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1424/2868] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:04.382643 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.393391 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a763eb-e9b1-48ad-ae64-1c48a06d8d68 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.398023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49089ce7-6765-4602-a425-ff62139eb694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.398620 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49089ce7-6765-4602-a425-ff62139eb694 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.398902 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49089ce7-6765-4602-a425-ff62139eb694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.399179 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49089ce7-6765-4602-a425-ff62139eb694 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.399817 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49089ce7-6765-4602-a425-ff62139eb694 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.401435 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49089ce7-6765-4602-a425-ff62139eb694 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.411365 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1445/2869] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:04.415600 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.426006 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf36af6-cb99-4c52-a8dc-337f2116436e admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.430625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.431269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.431550 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.431823 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] PATH_INFO: `/v3/projects/32f8dcdeef934a5889c183ec781bf613/users/dde185771ef242609a7e34d79994e111/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.446955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32f8dcdeef934a5889c183ec781bf613, user_id=dde185771ef242609a7e34d79994e111, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.449629 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.462170 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1425/2870] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:30:04 2022] PUT /identity/v3/projects/32f8dcdeef934a5889c183ec781bf613/users/dde185771ef242609a7e34d79994e111/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:04.526088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077dede0-136a-41b9-a523-303689802e26 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.526385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077dede0-136a-41b9-a523-303689802e26 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.526687 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077dede0-136a-41b9-a523-303689802e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.554800 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-077dede0-136a-41b9-a523-303689802e26 None None] MFA Rules not processed for user `dde185771ef242609a7e34d79994e111`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:04.556010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-077dede0-136a-41b9-a523-303689802e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:04.619867 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1446/2871] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:04.624318 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.684701 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986448e6-a4c0-44db-af70-0316f566f9b6 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.689909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.690516 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.690792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.691060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.691690 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.693626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.719242 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1426/2872] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:30:04.723470 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077dede0-136a-41b9-a523-303689802e26 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.733766 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077dede0-136a-41b9-a523-303689802e26 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.788487 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.789257 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.789543 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.789851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.790488 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.792400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.815032 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1447/2873] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 01 04:30:04.819032 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.829241 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638576b-03fc-4cf1-92d4-cc6acc4f9a9c admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.833919 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.834517 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.834797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.835069 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.835708 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.837341 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.848948 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1427/2874] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:04.852688 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.863180 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddd6868-88ac-463f-b8f0-c32c16b8b475 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.867715 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19673431-395c-40be-9f4a-c94d044aa18b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.868297 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19673431-395c-40be-9f4a-c94d044aa18b admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.868574 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19673431-395c-40be-9f4a-c94d044aa18b admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.868843 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19673431-395c-40be-9f4a-c94d044aa18b admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.869502 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19673431-395c-40be-9f4a-c94d044aa18b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.871064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19673431-395c-40be-9f4a-c94d044aa18b admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.880694 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1448/2875] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:04 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:04.884605 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:04.894792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3607a7e5-db54-45cf-a210-1a2c5a835116 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:04.899525 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:04.900164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.900485 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.900755 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] PATH_INFO: `/v3/projects/33de398a68e543b89e051b05b8920820/users/8bdc49866c124398bd972ee1fa01d9ba/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.915871 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=33de398a68e543b89e051b05b8920820, user_id=8bdc49866c124398bd972ee1fa01d9ba, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:04.918527 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:04.927514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:DELETE=6,keystone:SELECT=89,keystone:INSERT=5 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:04.932878 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1428/2876] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:30:04 2022] PUT /identity/v3/projects/33de398a68e543b89e051b05b8920820/users/8bdc49866c124398bd972ee1fa01d9ba/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:04.942989 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28ca11f-adb2-401c-b5f6-c659e1a5196a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:04.943213 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28ca11f-adb2-401c-b5f6-c659e1a5196a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:04.943539 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28ca11f-adb2-401c-b5f6-c659e1a5196a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:04.971088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-f28ca11f-adb2-401c-b5f6-c659e1a5196a None None] MFA Rules not processed for user `8bdc49866c124398bd972ee1fa01d9ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:04.972288 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f28ca11f-adb2-401c-b5f6-c659e1a5196a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:04.976132 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:05.037532 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1449/2877] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:05.042008 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:05.101504 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3a3c7-f1d9-420a-a220-c7cde1de0f1b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:05.106041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:05.106633 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.106941 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.107214 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.107832 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:05.109726 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:05.135739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1429/2878] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:30:05 2022] POST /identity/v3/projects => generated 383 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:30:05.139821 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28ca11f-adb2-401c-b5f6-c659e1a5196a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:05.150877 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28ca11f-adb2-401c-b5f6-c659e1a5196a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:05.205639 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:05.206183 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.206484 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.206756 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] PATH_INFO: `/v3/users` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.207364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:05.209300 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:05.230854 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1450/2879] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:30:05 2022] POST /identity/v3/users => generated 298 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:30:05.234927 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:05.245269 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7817813-5234-429c-825c-9c7d7ac49eb2 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:05.250154 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:05.250750 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.251035 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.251314 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.251989 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:05.253616 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:05.264127 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1430/2880] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:05.269088 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:05.278791 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9511ae8-5199-4996-82c7-95a3058eb860 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:05.283952 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:05.284593 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.284874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.285178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] PATH_INFO: `/v3/projects/f285e38b167842cab9a3956e5489e621/users/6b03ea596d884c3c889328ccbce2023f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.299346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f285e38b167842cab9a3956e5489e621, user_id=6b03ea596d884c3c889328ccbce2023f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:05.302433 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:05.314459 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1451/2881] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:30:05 2022] PUT /identity/v3/projects/f285e38b167842cab9a3956e5489e621/users/6b03ea596d884c3c889328ccbce2023f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:05.318446 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:05.328145 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000dba22-bae4-4507-b41c-0c9044278a2b admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:05.386719 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0352d43-a5d9-4ff2-b899-ffc065de22fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:05.387326 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0352d43-a5d9-4ff2-b899-ffc065de22fb admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.387610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0352d43-a5d9-4ff2-b899-ffc065de22fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.387887 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0352d43-a5d9-4ff2-b899-ffc065de22fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.388530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0352d43-a5d9-4ff2-b899-ffc065de22fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:05.390188 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0352d43-a5d9-4ff2-b899-ffc065de22fb admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:05.404200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1431/2882] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:05 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:05.405751 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:05.416833 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6164c276-6003-4596-bf4e-bc90b5b8f1b9 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:05.421458 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:05.422107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.422389 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.422665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] PATH_INFO: `/v3/projects/f285e38b167842cab9a3956e5489e621/users/6b03ea596d884c3c889328ccbce2023f/roles/1aa2abd4ab14431f8c1b50c02ea27881` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.424951 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f285e38b167842cab9a3956e5489e621, user_id=6b03ea596d884c3c889328ccbce2023f, role_id=1aa2abd4ab14431f8c1b50c02ea27881)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:05.427605 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:05.437627 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1452/2883] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:30:05 2022] PUT /identity/v3/projects/f285e38b167842cab9a3956e5489e621/users/6b03ea596d884c3c889328ccbce2023f/roles/1aa2abd4ab14431f8c1b50c02ea27881 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:05.449102 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.449420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.449699 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.478014 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] MFA Rules not processed for user `6b03ea596d884c3c889328ccbce2023f`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:05.479144 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:05.538588 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:05.549342 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1432/2884] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:05 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:05.560759 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:05.562520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:05.621654 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56387cfa-8667-469d-ba43-2f759e93adb6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:05.625441 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16340831-0425-4937-9088-571e0f90949c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:05.626037 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16340831-0425-4937-9088-571e0f90949c service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:05.626314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16340831-0425-4937-9088-571e0f90949c service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:05.626586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16340831-0425-4937-9088-571e0f90949c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:05.635497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:05.683010 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16340831-0425-4937-9088-571e0f90949c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:05.684445 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16340831-0425-4937-9088-571e0f90949c service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:05.694062 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1453/2885] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:07.215761 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:07.217238 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:07.229084 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb58c17-f56d-4193-aca6-c58a4c3354bf None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:07.232612 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:07.233245 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:07.233530 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:07.233806 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:07.290556 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:07.291907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:07.300730 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1433/2886] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:07.897302 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16340831-0425-4937-9088-571e0f90949c service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:07.905990 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16340831-0425-4937-9088-571e0f90949c service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:07.910322 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ad90ce-2275-4d93-ae1f-d910fa4f50fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:07.910911 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ad90ce-2275-4d93-ae1f-d910fa4f50fd service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:07.911187 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ad90ce-2275-4d93-ae1f-d910fa4f50fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:07.911454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ad90ce-2275-4d93-ae1f-d910fa4f50fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:07.920222 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ad90ce-2275-4d93-ae1f-d910fa4f50fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:07.921665 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ad90ce-2275-4d93-ae1f-d910fa4f50fd service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:07.931616 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1454/2887] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:07 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:11.814359 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:11.876955 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923d01eb-9059-4cb2-9290-f21e8ec9208c service cinder] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:11.881436 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:11.882054 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:11.882333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:11.882602 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] PATH_INFO: `/v3/users/dde185771ef242609a7e34d79994e111` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:11.883706 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dde185771ef242609a7e34d79994e111)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:11.885673 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:11.932299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dde185771ef242609a7e34d79994e111'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:11.932970 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dde185771ef242609a7e34d79994e111'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:11.940756 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dde185771ef242609a7e34d79994e111'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:11.941465 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dde185771ef242609a7e34d79994e111'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:11.948637 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dde185771ef242609a7e34d79994e111'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:11.965870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1434/2888] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:30:11 2022] DELETE /identity/v3/users/dde185771ef242609a7e34d79994e111 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 01 04:30:11.970493 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:11.970961 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:11.971082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:11.990223 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:11.991426 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:12.049820 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1455/2889] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:12.059344 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:12.060724 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:12.075709 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:12.120461 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761d8e2b-03df-45a6-94ff-82636f0df1b8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:12.124041 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:12.124636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:12.124913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:12.125212 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:12.134557 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:12.135957 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:12.145513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1435/2890] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:12.422984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:12.425110 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:12.439371 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e96f9c-0965-4663-afa2-66825028d50a None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:12.444036 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:12.445553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:12.445876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:12.446150 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] PATH_INFO: `/v3/projects/32f8dcdeef934a5889c183ec781bf613` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:12.447284 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32f8dcdeef934a5889c183ec781bf613)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:12.449271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:12.494823 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32f8dcdeef934a5889c183ec781bf613', 'request_id': 'req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.506394 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32f8dcdeef934a5889c183ec781bf613', 'request_id': 'req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.508861 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32f8dcdeef934a5889c183ec781bf613', 'request_id': 'req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.524553 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] The token cache is being invalidate because project 32f8dcdeef934a5889c183ec781bf613 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:30:12.524847 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.525524 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.526998 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1456/2891] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:30:12 2022] DELETE /identity/v3/projects/32f8dcdeef934a5889c183ec781bf613 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:12.530986 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:12.540441 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f23ae7e-da25-4447-850c-3d598e87744d service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:12.595254 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:12.595791 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:12.596074 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:12.596352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] PATH_INFO: `/v3/users/8bdc49866c124398bd972ee1fa01d9ba` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:12.597499 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8bdc49866c124398bd972ee1fa01d9ba)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:12.599405 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:12.645159 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8bdc49866c124398bd972ee1fa01d9ba'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.645804 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8bdc49866c124398bd972ee1fa01d9ba'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.652444 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8bdc49866c124398bd972ee1fa01d9ba'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.653093 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8bdc49866c124398bd972ee1fa01d9ba'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.660193 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8bdc49866c124398bd972ee1fa01d9ba'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:12.678112 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1436/2892] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:30:12 2022] DELETE /identity/v3/users/8bdc49866c124398bd972ee1fa01d9ba => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:12.956296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:12.957787 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:12.970444 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf0a5e-f9ca-45d1-965b-302b1bca28c8 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:12.996271 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:13.037550 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:13.038185 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:13.038461 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:13.038732 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] PATH_INFO: `/v3/projects/33de398a68e543b89e051b05b8920820` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:13.039827 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33de398a68e543b89e051b05b8920820)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:13.041745 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:13.086788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33de398a68e543b89e051b05b8920820', 'request_id': 'req-b46bdb3e-727a-475e-a7ea-8d487a147435'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.093248 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33de398a68e543b89e051b05b8920820', 'request_id': 'req-b46bdb3e-727a-475e-a7ea-8d487a147435'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.100084 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33de398a68e543b89e051b05b8920820', 'request_id': 'req-b46bdb3e-727a-475e-a7ea-8d487a147435'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.110072 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] The token cache is being invalidate because project 33de398a68e543b89e051b05b8920820 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:30:13.110964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.111649 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.113159 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1457/2893] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:30:12 2022] DELETE /identity/v3/projects/33de398a68e543b89e051b05b8920820 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:13.117384 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:13.118784 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:13.131695 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76783c51-1bea-4448-b4d9-86e3fdc792b0 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:13.163892 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:13.203938 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:13.204686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:13.205045 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:13.205389 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] PATH_INFO: `/v3/users/6b03ea596d884c3c889328ccbce2023f` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:13.206560 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b03ea596d884c3c889328ccbce2023f)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:13.208636 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:13.259432 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b03ea596d884c3c889328ccbce2023f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.260088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b03ea596d884c3c889328ccbce2023f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.267270 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b03ea596d884c3c889328ccbce2023f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.267918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b03ea596d884c3c889328ccbce2023f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.276946 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-85578cbc-f14e-4898-944d-4efd516d2475 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b03ea596d884c3c889328ccbce2023f'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.296964 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1437/2894] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:30:13 2022] DELETE /identity/v3/users/6b03ea596d884c3c889328ccbce2023f => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:13.572542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:13.574107 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:13.587400 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46bdb3e-727a-475e-a7ea-8d487a147435 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:13.645388 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:13.646020 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:13.646296 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:13.646567 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] PATH_INFO: `/v3/projects/f285e38b167842cab9a3956e5489e621` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:13.647661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f285e38b167842cab9a3956e5489e621)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:13.649591 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:13.694240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f285e38b167842cab9a3956e5489e621', 'request_id': 'req-6f9fdc48-9872-47bf-926b-8a1ef902ed89'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.697749 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:13.701849 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f285e38b167842cab9a3956e5489e621', 'request_id': 'req-6f9fdc48-9872-47bf-926b-8a1ef902ed89'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.709920 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f285e38b167842cab9a3956e5489e621', 'request_id': 'req-6f9fdc48-9872-47bf-926b-8a1ef902ed89'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.720975 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] The token cache is being invalidate because project f285e38b167842cab9a3956e5489e621 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:30:13.721270 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.721921 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-6f9fdc48-9872-47bf-926b-8a1ef902ed89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:30:13.723402 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1458/2895] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:30:13 2022] DELETE /identity/v3/projects/f285e38b167842cab9a3956e5489e621 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:13.729361 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76acb58d-c6a7-47e9-b18c-206ce3344256 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:13.729646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76acb58d-c6a7-47e9-b18c-206ce3344256 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:13.729918 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76acb58d-c6a7-47e9-b18c-206ce3344256 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:13.749317 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-76acb58d-c6a7-47e9-b18c-206ce3344256 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:13.750523 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-76acb58d-c6a7-47e9-b18c-206ce3344256 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:13.810088 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1438/2896] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:13.818753 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0623a6e-551c-48e9-87d3-4a3b65cc2107 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:13.819042 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0623a6e-551c-48e9-87d3-4a3b65cc2107 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:13.819325 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0623a6e-551c-48e9-87d3-4a3b65cc2107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:13.839876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-e0623a6e-551c-48e9-87d3-4a3b65cc2107 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:13.841116 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-e0623a6e-551c-48e9-87d3-4a3b65cc2107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:13.848896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1459/2897] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:13.853164 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76acb58d-c6a7-47e9-b18c-206ce3344256 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:13.862819 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76acb58d-c6a7-47e9-b18c-206ce3344256 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:13.867285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:13.867870 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:13.868148 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:13.868417 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:13.869095 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:30:13.869413 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:13.870978 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:13.884183 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1439/2898] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:30:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:30:13.888416 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0623a6e-551c-48e9-87d3-4a3b65cc2107 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:13.898702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0623a6e-551c-48e9-87d3-4a3b65cc2107 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:13.903364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:13.903991 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:13.904274 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:13.904548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:13.905422 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:13.907497 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:13.933768 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1460/2899] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:30:13 2022] POST /identity/v3/projects => generated 381 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:30:13.937875 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:13.948248 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ff586-8796-4174-b144-7cbc16f153d1 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:13.987822 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:14.005848 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:14.006452 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:14.006734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:14.007010 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:14.007647 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:14.009598 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:14.030990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1440/2900] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:30:13 2022] POST /identity/v3/users => generated 291 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:30:14.035382 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:14.045716 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c441103-c507-439c-9797-d7b138e4c184 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:14.050563 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:14.051164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:14.051443 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:14.051714 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:14.052344 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:14.053964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:14.064645 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1461/2901] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:14.068877 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:14.079651 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e24fea3-5700-4143-8b14-ccfa6fbb1ab8 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:14.084289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9dbaf-546d-43d0-999a-fe08d158351b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:14.084883 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9dbaf-546d-43d0-999a-fe08d158351b admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:14.085200 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9dbaf-546d-43d0-999a-fe08d158351b admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:14.085470 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9dbaf-546d-43d0-999a-fe08d158351b admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:14.086108 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9dbaf-546d-43d0-999a-fe08d158351b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:14.087678 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9dbaf-546d-43d0-999a-fe08d158351b admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:14.099050 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1441/2902] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:30:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:30:14.103298 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:14.113560 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2613e9de-abd0-468a-9b1a-46f44e8378e6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:14.118153 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:14.118803 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:14.119085 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:14.119360 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] PATH_INFO: `/v3/projects/c465c8ca23c943ab87092a9544383677/users/e27ccd9a0974420389d2216ca0590082/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:14.134359 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c465c8ca23c943ab87092a9544383677, user_id=e27ccd9a0974420389d2216ca0590082, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:14.137038 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:14.150955 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1462/2903] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:30:14 2022] PUT /identity/v3/projects/c465c8ca23c943ab87092a9544383677/users/e27ccd9a0974420389d2216ca0590082/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:30:14.155990 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e83414-1a4f-428a-9201-140d1b448b1b None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:14.156451 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e83414-1a4f-428a-9201-140d1b448b1b None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:14.156570 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e83414-1a4f-428a-9201-140d1b448b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:14.175780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-56e83414-1a4f-428a-9201-140d1b448b1b None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:14.177013 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-56e83414-1a4f-428a-9201-140d1b448b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:14.237015 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1442/2904] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:14.247995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:14.250945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:14.309675 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24900c0-0338-4ec3-b051-557cff92f949 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:14.313908 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:14.314520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:14.314796 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:14.315069 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:14.323896 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:14.325364 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:14.334501 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1463/2905] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:14.950764 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb4920a-00e9-45c5-b481-4168515ea52c None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:14.951169 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb4920a-00e9-45c5-b481-4168515ea52c None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:14.951580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb4920a-00e9-45c5-b481-4168515ea52c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:14.983299 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-fbb4920a-00e9-45c5-b481-4168515ea52c None None] MFA Rules not processed for user `e27ccd9a0974420389d2216ca0590082`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:14.984638 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fbb4920a-00e9-45c5-b481-4168515ea52c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:15.052150 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1443/2906] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:14 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:15.064427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:15.065842 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:15.127460 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e565ce-88ae-4e7b-bda7-48a18802257f service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:15.131000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:15.131602 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:15.131876 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:15.132148 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:15.142668 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:15.144384 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:15.153454 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1464/2907] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:30:16.515825 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:16.516991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:16.516991 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:16.536304 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] MFA Rules not processed for user `e27ccd9a0974420389d2216ca0590082`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:30:16.537548 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:16.544513 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1444/2908] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:30:16 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:30:16.778427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:16.779894 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:16.792385 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ee3b1f-31ad-41c6-9a0b-86bcfaa15ff0 service cinder] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:16.796011 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:16.796548 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:16.796828 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:16.797137 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:16.799449 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:16.808758 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:16.810281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:16.818794 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1465/2909] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:17.131798 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:17.133333 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:17.184518 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:17.214301 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc5eda-bbef-419f-9f2b-f0e393071413 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:17.219208 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:17.220455 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:17.220731 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:17.221028 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:17.222739 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:17.234060 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:17.235774 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:17.245610 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1445/2910] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:18.889515 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:18.899095 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3746bec5-e6ec-4a14-b092-d5807cf916da service neutron] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:18.902869 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d59641-2b3e-490c-96f3-3431217324ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:18.903610 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d59641-2b3e-490c-96f3-3431217324ef service neutron] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:18.903964 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d59641-2b3e-490c-96f3-3431217324ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:18.904983 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d59641-2b3e-490c-96f3-3431217324ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:18.906425 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-28d59641-2b3e-490c-96f3-3431217324ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:18.917914 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d59641-2b3e-490c-96f3-3431217324ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:18.919906 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d59641-2b3e-490c-96f3-3431217324ef service neutron] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:18.924082 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:18.930835 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1466/2911] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:24.046884 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:30:24.050660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:24.136059 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426842cd-4594-48db-93e7-f12cf283e581 service placement] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:30:24.141780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '901b9bb47c2f40c2b1b4cfe7886a1167', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:30:24.142493 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:30:24.142773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:30:24.143047 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:30:24.145103 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:30:24.158233 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:30:24.159820 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:30:24.170769 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1446/2912] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:30:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:30:28.933874 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:30:34.166907 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=46 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:31:15.526790 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9aac6f-a4eb-41c4-8bbe-1cd1eb8b9095 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:31:15.527239 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9aac6f-a4eb-41c4-8bbe-1cd1eb8b9095 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:31:15.527368 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9aac6f-a4eb-41c4-8bbe-1cd1eb8b9095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:31:15.546237 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-1b9aac6f-a4eb-41c4-8bbe-1cd1eb8b9095 None None] MFA Rules not processed for user `e27ccd9a0974420389d2216ca0590082`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:31:15.547411 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-1b9aac6f-a4eb-41c4-8bbe-1cd1eb8b9095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:31:15.554066 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1467/2913] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:31:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:31:20.598738 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:31:20.600121 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:31:20.613911 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132744dc-9f7b-43a8-b31a-6a84e32dad63 service nova] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:31:20.618814 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:31:20.619406 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:31:20.619688 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:31:20.619963 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:31:20.621679 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:31:20.633388 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:31:20.634973 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:31:20.645827 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1447/2914] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:31:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:31:25.541363 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:31:30.641331 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=6 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:32:12.811815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9aac6f-a4eb-41c4-8bbe-1cd1eb8b9095 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:12.824293 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9aac6f-a4eb-41c4-8bbe-1cd1eb8b9095 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:12.829392 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:12.830023 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:12.830851 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:12.831552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:12.833426 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:12.845968 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:12.848112 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:12.859280 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1468/2915] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:32:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 01 04:32:13.072845 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.083916 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7956f0-9199-4fa5-9b4e-d9bf50dd0d24 service placement] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.088377 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.089037 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.089349 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.089580 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] PATH_INFO: `/v3/users/e27ccd9a0974420389d2216ca0590082` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.090683 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e27ccd9a0974420389d2216ca0590082)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.092654 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.139179 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e27ccd9a0974420389d2216ca0590082'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.139905 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e27ccd9a0974420389d2216ca0590082'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.147058 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e27ccd9a0974420389d2216ca0590082'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.147685 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e27ccd9a0974420389d2216ca0590082'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.154024 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-f4930d76-1ce6-4452-91ec-b88c94d5b5c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e27ccd9a0974420389d2216ca0590082'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.171660 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1448/2916] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:32:13 2022] DELETE /identity/v3/users/e27ccd9a0974420389d2216ca0590082 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:32:13.351027 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.363596 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51edf46-9893-4e1d-8582-5aa7a85f1ab0 service placement] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.420782 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.421434 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.421712 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.421982 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] PATH_INFO: `/v3/projects/c465c8ca23c943ab87092a9544383677` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.423156 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c465c8ca23c943ab87092a9544383677)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.425212 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.469824 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c465c8ca23c943ab87092a9544383677', 'request_id': 'req-9628f909-972b-44f5-b4b4-8b28c7b22491'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.478520 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c465c8ca23c943ab87092a9544383677', 'request_id': 'req-9628f909-972b-44f5-b4b4-8b28c7b22491'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.484764 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c465c8ca23c943ab87092a9544383677', 'request_id': 'req-9628f909-972b-44f5-b4b4-8b28c7b22491'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.495167 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] The token cache is being invalidate because project c465c8ca23c943ab87092a9544383677 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:32:13.495552 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.496178 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-9628f909-972b-44f5-b4b4-8b28c7b22491 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:13.497661 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1469/2917] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:32:13 2022] DELETE /identity/v3/projects/c465c8ca23c943ab87092a9544383677 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:32:13.503107 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0d03e-1bc3-4325-a55c-dc9f2765fc74 None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.503403 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0d03e-1bc3-4325-a55c-dc9f2765fc74 None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.503686 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0d03e-1bc3-4325-a55c-dc9f2765fc74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.527229 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-0cb0d03e-1bc3-4325-a55c-dc9f2765fc74 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:32:13.528509 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-0cb0d03e-1bc3-4325-a55c-dc9f2765fc74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:13.589780 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1449/2918] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:32:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:32:13.598064 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ddfcf8-a771-4106-8fb8-a8df959113fd None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.598346 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ddfcf8-a771-4106-8fb8-a8df959113fd None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.598618 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ddfcf8-a771-4106-8fb8-a8df959113fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.619705 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-33ddfcf8-a771-4106-8fb8-a8df959113fd None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:32:13.620845 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-33ddfcf8-a771-4106-8fb8-a8df959113fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:13.629635 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1470/2919] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:32:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 01 04:32:13.633811 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb0d03e-1bc3-4325-a55c-dc9f2765fc74 None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.644457 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb0d03e-1bc3-4325-a55c-dc9f2765fc74 None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.649077 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.649672 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.649953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.650225 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] PATH_INFO: `/v3/domains` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.650869 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81586) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 01 04:32:13.651171 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.652747 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.665422 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1450/2920] 10.222.0.26 () {58 vars in 1153 bytes} [Sat Oct 1 04:32:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 01 04:32:13.669717 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ddfcf8-a771-4106-8fb8-a8df959113fd None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.678815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ddfcf8-a771-4106-8fb8-a8df959113fd None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.683463 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.684055 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.684335 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.684606 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] PATH_INFO: `/v3/projects` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.685277 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.687164 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.714806 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1471/2921] 10.222.0.26 () {60 vars in 1153 bytes} [Sat Oct 1 04:32:13 2022] POST /identity/v3/projects => generated 397 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:32:13.718909 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.729009 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9496d7-d567-4e0b-9f52-fe17370780cc admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.786898 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.787514 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.787797 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.788072 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] PATH_INFO: `/v3/users` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.788703 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.790648 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.798533 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:32:13.813773 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1451/2922] 10.222.0.26 () {60 vars in 1144 bytes} [Sat Oct 1 04:32:13 2022] POST /identity/v3/users => generated 299 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 01 04:32:13.818984 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.831742 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9782ac8-27e4-4431-86a9-d3a339e6803f admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.836788 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.837545 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.837822 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.838106 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.838718 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.840308 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.851044 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1472/2923] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:32:13 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:32:13.855285 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.865420 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211d3783-6ec4-4dd2-9434-60df83a2fa13 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.870048 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5373daa7-3b37-4f1e-919b-38781dbee6a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.870582 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5373daa7-3b37-4f1e-919b-38781dbee6a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.870860 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5373daa7-3b37-4f1e-919b-38781dbee6a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.871130 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5373daa7-3b37-4f1e-919b-38781dbee6a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.871792 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5373daa7-3b37-4f1e-919b-38781dbee6a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.873418 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5373daa7-3b37-4f1e-919b-38781dbee6a9 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.884571 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1452/2924] 10.222.0.26 () {58 vars in 1122 bytes} [Sat Oct 1 04:32:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 01 04:32:13.888667 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:13.899347 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5cbb2-c3e6-46ff-9c7e-5335fe5090e6 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:13.903872 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:13.904507 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] REQUEST_METHOD: `PUT` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.904786 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.905096 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] PATH_INFO: `/v3/projects/1524c57135e64352972df7b24140b2fb/users/2364ea1d696c4bfabb11fe2f3bc43d5d/roles/d261fc9497d44fedb05d9b45875771be` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.919281 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1524c57135e64352972df7b24140b2fb, user_id=2364ea1d696c4bfabb11fe2f3bc43d5d, role_id=d261fc9497d44fedb05d9b45875771be)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:13.921995 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:13.934798 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1473/2925] 10.222.0.26 () {60 vars in 1483 bytes} [Sat Oct 1 04:32:13 2022] PUT /identity/v3/projects/1524c57135e64352972df7b24140b2fb/users/2364ea1d696c4bfabb11fe2f3bc43d5d/roles/d261fc9497d44fedb05d9b45875771be => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:32:13.946408 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02daa15-f426-4adb-88b6-a1e4474f530a None None] REQUEST_METHOD: `POST` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:13.946718 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02daa15-f426-4adb-88b6-a1e4474f530a None None] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:13.946992 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02daa15-f426-4adb-88b6-a1e4474f530a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:13.974307 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.auth.core [None req-b02daa15-f426-4adb-88b6-a1e4474f530a None None] MFA Rules not processed for user `2364ea1d696c4bfabb11fe2f3bc43d5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81586) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:32:13.975494 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-b02daa15-f426-4adb-88b6-a1e4474f530a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:14.037140 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1453/2926] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:32:13 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:32:14.048231 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:14.050170 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:14.064078 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:32:14.109000 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba21aaa-8e2c-406a-afeb-3c5431193856 admin admin] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:14.113586 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e58f93f-abe1-415a-8b61-bf36ea713332 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '52b5d64c999a4f16afec678df3025529', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:14.114189 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58f93f-abe1-415a-8b61-bf36ea713332 service cinder] REQUEST_METHOD: `GET` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:14.114468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58f93f-abe1-415a-8b61-bf36ea713332 service cinder] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:14.114738 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58f93f-abe1-415a-8b61-bf36ea713332 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:14.123240 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e58f93f-abe1-415a-8b61-bf36ea713332 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:14.124601 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e58f93f-abe1-415a-8b61-bf36ea713332 service cinder] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:14.136892 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1474/2927] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:32:14 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:32:21.193239 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02daa15-f426-4adb-88b6-a1e4474f530a None None] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:21.254532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02daa15-f426-4adb-88b6-a1e4474f530a None None] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:21.259625 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:21.260258 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:21.260540 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:21.260818 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] PATH_INFO: `/v3/users/2364ea1d696c4bfabb11fe2f3bc43d5d` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:21.262005 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2364ea1d696c4bfabb11fe2f3bc43d5d)` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:21.263928 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:21.309734 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2364ea1d696c4bfabb11fe2f3bc43d5d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.310372 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2364ea1d696c4bfabb11fe2f3bc43d5d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.317186 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2364ea1d696c4bfabb11fe2f3bc43d5d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.317808 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2364ea1d696c4bfabb11fe2f3bc43d5d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.324705 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.notifications [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2364ea1d696c4bfabb11fe2f3bc43d5d'} {{(pid=81586) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.329961 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:INSERT=4,keystone:SELECT=87,keystone:DELETE=9 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:32:21.343626 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1454/2928] 10.222.0.26 () {58 vars in 1224 bytes} [Sat Oct 1 04:32:21 2022] DELETE /identity/v3/users/2364ea1d696c4bfabb11fe2f3bc43d5d => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:32:21.349079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] REQUEST_METHOD: `POST` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:21.349079 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:21.349690 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:21.369126 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.auth.core [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] MFA Rules not processed for user `280e8a1dffee4eacad394696d3ff53d3`. Rule list: `[]` (Enabled: `True`). {{(pid=81587) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 01 04:32:21.370314 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:21.427791 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1475/2929] 10.222.0.26 () {58 vars in 958 bytes} [Sat Oct 1 04:32:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 01 04:32:21.437555 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:21.438913 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:21.495371 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a870d7-cc2f-4923-8242-fca72719b121 admin admin] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:21.498953 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '16aab142c0a7445483caf19178d56062', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:21.499542 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:21.499821 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:21.500089 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:21.509289 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:21.510646 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:21.520508 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1455/2930] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:32:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:32:21.785542 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] Authenticating user token {{(pid=81587) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:32:21.786957 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.fernet_utils [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81587) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:32:21.798673 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd55815-60dd-4536-90d1-b9824cdc6597 None None] Validating token access rules against request {{(pid=81587) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:32:21.803702 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '280e8a1dffee4eacad394696d3ff53d3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '49c20dd6d2894e7799f891fe5467c7e0', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81587) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:32:21.804381 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:32:21.804662 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:32:21.804945 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] PATH_INFO: `/v3/projects/1524c57135e64352972df7b24140b2fb` {{(pid=81587) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:32:21.806077 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=1524c57135e64352972df7b24140b2fb)` {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:32:21.808007 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] RBAC: Authorization granted {{(pid=81587) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:32:21.854947 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1524c57135e64352972df7b24140b2fb', 'request_id': 'req-61449a83-9e68-4eab-a68f-32fa6b0c94ff'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.861630 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1524c57135e64352972df7b24140b2fb', 'request_id': 'req-61449a83-9e68-4eab-a68f-32fa6b0c94ff'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.868028 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1524c57135e64352972df7b24140b2fb', 'request_id': 'req-61449a83-9e68-4eab-a68f-32fa6b0c94ff'} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.878530 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] The token cache is being invalidate because project 1524c57135e64352972df7b24140b2fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81587) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 01 04:32:21.878815 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.879427 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG keystone.notifications [None req-61449a83-9e68-4eab-a68f-32fa6b0c94ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81587) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 01 04:32:21.880875 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: [pid: 81587|app: 0|req: 1476/2931] 10.222.0.26 () {58 vars in 1233 bytes} [Sat Oct 1 04:32:21 2022] DELETE /identity/v3/projects/1524c57135e64352972df7b24140b2fb => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 01 04:32:31.515754 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=31,keystone:DELETE=1 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:32:31.875468 dv-cinder-860044-1-iscsi devstack@keystone.service[81587]: DEBUG dbcounter [-] [81587] Writing DB stats keystone:SELECT=74,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81587) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 01 04:37:06.968594 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] Authenticating user token {{(pid=81586) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 01 04:37:06.970110 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:37:07.107982 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365b5464-aa43-44b0-81a4-fafecb243ad8 service neutron] Validating token access rules against request {{(pid=81586) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 01 04:37:07.165482 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e158bca-6b88-460e-8434-9b9fa6e0b25d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4302238ce83b4591b3d062ef87517f1c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '469129ac7df14a70a9cb14818c437800', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81586) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 01 04:37:07.166075 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e158bca-6b88-460e-8434-9b9fa6e0b25d service placement] REQUEST_METHOD: `GET` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 01 04:37:07.166352 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e158bca-6b88-460e-8434-9b9fa6e0b25d service placement] SCRIPT_NAME: `/identity` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 01 04:37:07.166622 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e158bca-6b88-460e-8434-9b9fa6e0b25d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81586) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 01 04:37:07.168345 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.fernet_utils [None req-9e158bca-6b88-460e-8434-9b9fa6e0b25d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81586) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 01 04:37:07.179261 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e158bca-6b88-460e-8434-9b9fa6e0b25d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 01 04:37:07.181532 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e158bca-6b88-460e-8434-9b9fa6e0b25d service placement] RBAC: Authorization granted {{(pid=81586) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 01 04:37:07.191423 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: [pid: 81586|app: 0|req: 1456/2932] 10.222.0.26 () {60 vars in 1365 bytes} [Sat Oct 1 04:37:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 01 04:37:17.187173 dv-cinder-860044-1-iscsi devstack@keystone.service[81586]: DEBUG dbcounter [-] [81586] Writing DB stats keystone:SELECT=89 {{(pid=81586) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}